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Executive summary 


The project 


Switch-on is an intensive, targeted literacy intervention that aims to improve the reading skills of pupils 
who are struggling with literacy. There are two versions of the intervention: Switch-on Reading and 
Switch-on Reading and Writing. Both involve specially trained Teaching Assistants (TAs) delivering a 
tailored programme of literacy support in daily 20-minute sessions over a ten-week period. 


A previous EEF-funded evaluation of Switch-on had shown signs of promise in raising reading 
outcomes for Year 7 pupils, hence a larger scale effectiveness evaluation was conducted. Each of the 
184 participating schools were randomly assigned to receive either Switch-on Reading, Switch-on 
Reading and Writing, or continue their usual practices of supporting pupils with reading difficulties 
(called ‘business as usual’ control). Between two and four TAs in each school were trained to deliver 
the interventions by a team of trainers who were distinct from the developers. TAs received 1.5 days 
training for Switch-on Reading, and 2.5 days for Switch-on Reading and Writing. Each TA worked with 
between two and four Year 3 pupils who had not met age-related expectations in literacy at the end of 
Key Stage 1, and who did not have a high level of special needs. In total, 999 pupils were involved in 
the trial. The primary outcome measure was the Hodder Group Reading Test 2A. The evaluation was 
conducted over the 2015/2016 academic year. A process and implementation evaluation was carried 
out which included depth telephone interviews with the developers, trainers, and TAs, and six case 
studies incorporating the views of TAs, class teachers, and Switch-on co-ordinators in the school. 


Key conclusions 


1. Participating children in schools delivering either version of Switch-on made no additional 
progress in reading compared to similarly struggling children in ‘business as usual’ control 
schools. The 4 padlock security rating means that we have high confidence that there was no 
difference, and that this was due to Switch-on and not affected by other factors. 


2. A similar result was observed for children eligible for Free School Meals. 


3. The secondary analysis suggests that Switch-on might have affected children who did not 
actually receive the intervention by changing the make-up of their class or the capacity of their 
TAs. These findings are tentative, but emphasise the importance of considering potential 
impacts on all children when using targeted interventions. 


4. Overall, participating staff were positive about the intervention and accompanying training. 
Perceived outcomes for pupils included increased confidence, motivation and interest in 
reading, and improved reading and writing skills. Skills of participating TAs were also felt to have 
improved. 


5. Some schools reported modifications to the prescribed content, duration and format of Switch- 
on sessions. Closer monitoring by the developers could help to ensure greater fidelity in future. 
It was also reported that some class teachers had limited awareness of Switch-on. Clarifying 
the role of Switch-on as part of a broader, teacher-led literacy strategy could improve 
implementation. 


EEF security rating Security rating awarded as part of 
the EEF peer review process 

The security rating of the trial indicates how confident we can be that any additional progress 

experienced by the children receiving Switch-on was due to the intervention rather than any other 


factors. This trial was an effectiveness trial that aimed to test whether the intervention was effective 
when delivered in a form that could be available to a large number of schools. Findings from this study 
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have high security. The trial was a well-designed, three-arm randomised controlled trial, with 
randomisation at school level. Relatively few pupils were lost to the analysis and pupils who received 
the intervention were similar to the pupils in the comparison group. The trial was only designed to 
reliably detect an impact equivalent to three months’ progress. 


Additional findings 


The trial did not find evidence that a three-month delivery of Switch-on Reading or Switch-on Reading 
and Writing improves reading outcomes of pupils struggling with literacy at Key Stage 1 compared to 
schools’ usual practices. Estimated effects were at zero standard deviations and not statistically 
significant. The intervention also showed no effect on pupils eligible for free school meals. These 
findings contradict previous evidence from a smaller within-school efficacy trial and the literature on the 
effectiveness of structured, one-on-one literacy interventions led by TAs. 


The process evaluation suggests that in some schools intervention fidelity was compromised to some 
degree, which is likely to have affected the quality of the intervention and could explain the lack of 
impact. The process evaluation also indicated inconsistent involvement of class teachers and senior 
leaders, which is identified in the literature as a factor that enables effective implementation. 


The secondary analysis suggests that Switch-on might have affected children who didn’t actually 
receive the intervention by changing the make-up of their class or the capacity of their TAs. There is a 
possibility that implementing Switch-on may have had a negative impact on the literacy of pupils in the 
same year who were eligible to receive the intervention but were not selected due to limitations on 
numbers. Conversely, there is a possibility that the intervention had a positive impact for pupils in the 
same that class that were ineligible to receive the intervention. Both results should be treated cautiously. 


Cost 


The cost of the project was estimated at £546 per pupil. This estimate includes up-front costs (for 
website development and access, and for the development of training materials such as films) and 
ongoing delivery costs (including provision of books and TA and trainer training). In terms of staff time, 
approximately 68 hours per pupil were required to deliver Switch-on. This included training and the 
delivery of sessions. If the intervention were delivered in a school over a period of three years, the cost 
per pupil per year would be much lower, at £184, because most of the cost are up-front costs. Estimates 
are based on an average of five pupils per school and two or three TAs per school delivering the 
intervention. 


Table 1: Summary of impact on primary outcome 


Effect size Estimated 


oe months’ 
Coxe) avi le(=1aler=) ellale) 
Ta iae1)) ited 


EEF security EEF cost 


eT colt 0) Type of trial rating 


Switch-on vs. 0.00 ; ; 
(-0.13, 0.13) 8 elas 


Switch-on FSM vs. 0.03 
control (-0.18, 0.20) 


Effectiveness 


Introduction 


Switch-on: intervention description 


1. Rationale and aims 
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Switch-on is an intensive literacy intervention which was developed by Nottinghamshire Reading 
Recovery Teacher Leaders. The intervention was developed in response to requests from 
headteachers to provide TAs with professional development in literacy. The needs identified for 
improved training of TAs reflect findings in a wider review of evidence on ‘Making the Best Use of 
Teaching Assistants’ which found that ‘the typical deployment and use of TAs, under everyday 
conditions, is not leading to improvements in academic outcomes’ (Sharples et a/., 2015, p. 11). 


By providing TAs with structured support and training to deliver a one-to-one reading intervention, the 
aim of the intervention was to enable children: 


‘to participate more fully in the classroom by becoming more confident, active and independent 
readers, who can use a range of effective reading strategies in order to achieve their full reading 
potential’ (Nottinghamshire County Council Switch-on training manual, p. 2). 


The intervention manual and accompanying training introduced TAs to the following theoretical 
frameworks: 


e Vygotsky’s theory of the zone of proximal development, which draws a distinction between what 
a learner can do independently and with help from an adult or more competent peer 
(Smagorinsky, 2011); and 

e the ‘simple view of reading’, which distinguishes between language comprehension processes 
and word recognition skills and demonstrates that both are essential for effective reading 
(Gough and Tunmer, 1986). 


It also explained the rationale for re-reading familiar books and how this approach helped to build 
confidence and fluency. 


2. Who (recipients) 


The intervention was aimed at pupils in Year 3 (aged seven and eight) who were working below age- 
related expectations at the end of Key Stage 1 (five to seven years) and did not have a high level of 
special needs. Schools were able to select pupils who met these criteria—reflecting the need to reflect 
real-life choices and contexts in a large scale effectiveness trial—and were advised only to select pupils 
for whom, in their view, the intervention would be appropriate and beneficial. 


3. What (materials) 


TAs were provided with an intervention manual which included: 


e the rationale for the intervention; 

e the theory of reading progress underpinning the intervention; 

e detailed instructions about how the intervention should be delivered, including examples; and 

e forms to monitor and record the progress of pupils, and explanations of how these should be 
used. 


After the initial training, each school received a pack of 180 reading books to be used in the delivery of 
Switch-on. This bank of reading resources, from a variety of publishers, had an increasing gradient of 
challenge to meet the needs of all abilities. There was one book for each of the 30 reading levels across 
15 colour bands. Books were fictional narrative texts. 


4. What (procedures 


Two versions of the Switch-on programme were included in the trial: Switch-on Reading and Switch-on 
Reading and Writing. Switch-on Reading consisted of daily reading sessions and Switch-on Reading 
and Writing alternated between writing and reading sessions. 
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Key features of Switch-on and how it was intended to be delivered included: 


e It was designed to be interactive and to encourage active and independent problem- 
solving. 

e It was based on observation, monitoring of progress, and detailed feedback. TAs used 
‘Running Records’, a form of shorthand developed by Reading Recovery to record word by 
word pupils’ mistakes and successes when reading. They used this to ensure pupils were 
accessing appropriately challenging texts, to develop understanding of how the child was 
learning, and to provide feedback and tailored prompts. 

e It was intended to be distinct from classroom learning. This was emphasized in the writing 
sessions by the use of a different notebook and different pens. Switch-on was nonetheless 
intended to be embedded in pupils’ wider learning and classroom work through communication 
between the TA and class teacher. This was emphasised in trainers’ training. 


In terms of content, the breakdown of each session is summarised in Figure 1Error! Reference source 
not found.. 


Figure 1: Content of Switch-on Reading and Writing 


Switch-on Reading session Switch-on Writing session 


+ independent reading of two familiar books «independent reading of a familiar book; 
(books that have been read twice before); *talking about the story and choosing a 
* taking ‘running records’ on a book that was sentence to write; and 


introduced and read once before; * writing the sentence in the notebook with 
¢ introduction of a new book; and support. 
*reading a new book with support. 


Each session of Switch-on Reading incorporated the revision of at least one familiar text, the 
introduction of a new text and vocabulary, practising phonics, and some work on comprehension 
through questions and discussions. There was an expectation that in each session the pupil would read 
excerpts from four books. In Switch-on Writing, pupils read one book. 


In practice, some modifications were made to the regularity and the content of the sessions. This is 
explored in the Process Evaluation section of this report. 


5. Who (implementers) 


Switch-on was designed to be delivered by trained TAs. Schools were asked to allocate a minimum of 
two TAs to the intervention as well as a co-ordinator, who was responsible for supporting and 
overseeing delivery of the intervention in the school. It was specified that this should be a member of 
the senior management team. 


6. How (mode of delivery) 


Switch-on is a personalised one-to-one intervention that is tailored to the needs of individual pupils. It 
was delivered face to face. No specific guidance was included in the intervention manual about when 
sessions should take place and if they should be delivered during or outside of class time. 


7. Where (setting) 


Switch-on Effectiveness Trial 


The intervention took place in school. No specific guidance was included in the intervention manual 
about which settings within the school were most appropriate, leaving flexibility for schools to choose. 


8. When and how much (dosage) 


Delivery of Switch-on began in schools in the first week of the Spring term (week commencing 4 January 
2016) and included daily 20-minute sessions over a ten-week period. 


Figure 2: Timeframe of the intervention delivery 


Memorandum of Understanding signed 


September 2015 Parental Opt-out consent obtained 


November - 


December 2015 Teaching Assistants training 


January 2016 Intervention starts in treatment schools 


Visit of Switch-on trainer to schools 


February 2016 Half-day training for TAs 


Intervention finishes 


March 2016 


April-May2016 


Pupil testing in all schools 


9. Tailoring 


Switch-on is a manualised intervention and optimal treatment fidelity is emphasised. Nonetheless, TAs 
were encouraged to adapt their support strategies to the specific needs of pupils. 


10. How well (planned) 


A ‘train the trainer’ model was adopted in order to maximise implementation effectiveness. In this model 
the Switch-on developers trained 21 Switch-on trainers, who then went on to train TAs in the 
participating schools. In October 2015, the Switch-on trainers received two days (14 hours) training on 
how to deliver Switch-on Reading and Writing. They trained TAs in November and December 2015. 


TAs and school coordinators both attended training on how to deliver Switch-on in their schools. Those 
in the Switch-on Reading group received one day of training, while those in the Reading and Writing 
group received two days (fourteen hours), one day each on reading and writing. The training provided 
background information on Switch-on, including the rationale for the content and approach, and also 
included demonstration of the use of the running record used to log the individual pupil’s progress. 
There was also a video demonstration of the key components of a Switch-on session. 


Half way through the programme (at approximately five weeks), TAs received a further half-day training 
session. Trainers also made one visit to TAs in their schools to observe delivery and provide feedback 
as well as answer any questions. Switch-on trainers also offered technical support and assistance by 
phone and email. 


Switch-on Effectiveness Trial 


Background evidence 


A previous efficacy trial evaluated the reading element of the full Switch-on programme. This trial was 
funded by the EEF as part of a funding round dedicated to reading catch-up projects for children at the 
transition from primary to secondary school. The writing component of Switch-on was not included. The 
earlier efficacy trial involved Year 7 pupils who had not reached a secure level 4 at the end of Key Stage 
2. It found that the programme ‘made a noticeable positive impact’ (Gorard ef al., 2014, p. 5). The 
findings suggested that, on average, pupils receiving the intervention would make approximately three 
additional months’ progress over the course of a term compared with pupils who did not receive the 
intervention. In particular, the intervention was shown to have a positive impact on the literacy learning 
of pupils with special educational needs (SEN), those in receipt of pupil premium, and lower attainers 
(Gorard et. al., 2014, p. 5). 


This effectiveness trial sought to test the findings from the efficacy trial on a larger sample of schools 
using a cascaded ‘training the trainer’ approach. This more accurately reflected the conditions required 
for a larger scale-up. It also included Switch-on Reading and Writing, as well as Switch-on Reading. 
The effectiveness trial included pupils in Year 3 rather than Year 7, reflecting the EEF’s focus on early 
intervention and prevention. In addition, based on their experience of delivering Switch-on, the 
developers considered this to be the age group most likely to benefit from the intervention. 


Evaluation objectives 


The trial consisted of an impact study and a process study. The impact evaluation sought to establish 
whether, and to what extent, Switch-on improved outcomes in reading. The trial was an effectiveness 
trial, focused on determining how the intervention performs in the circumstances in which schools would 
normally be implementing it (that is, at scale). 


There were three principal research questions tested by the project: 


1. What is the impact of Switch-on Reading and Switch-on Reading and Writing on reading 
outcomes for eligible Year 3 children? 

2. Does Switch-on Reading and Writing achieve better outcomes that Switch-on Reading? 

3. Are there (positive or negative) spill-over effects on non-participating children that have reading 
difficulties, or on non-participating children? 


The main aim of the process evaluation was to identify the main enablers and barriers to effective 
scaling-up of delivery of Switch-on. 


The research protocol can be found here: 


https://educationendowmentfoundation.org.uk/our-work/projects/switch-on-effectiveness-trial/ 
Ethical review 


NatCen Social Research obtained ethical approval from its internal Research Ethics Committee for 
the evaluation. Approval was sought for the opt-out process, communications and interviews with 
TAs, class teachers, co-ordinators, and Switch-on trainers, as well as the pupil testing, data linkage, 
and analysis of test results. Approval was granted on 31 July 2015. 


Project team 


The team responsible for delivering Switch-on consisted of: 


Paula Burrell, Intervention Developer 
Jose Coles, Intervention Developer 
Switch-on Trainers (21) 
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Teaching Assistants (297) 


NatCen Social Research were the independent evaluators. The research team was led by Rakhee 
Patel and the evaluation design lead was Nico Jabin. They were supported by: 


Loraine Bussard, Researcher 

Javiera Cartagena, Senior Researcher 

Sarah Haywood, Senior Researcher (now at the Department for Work and Pensions) 
Michael Lumpkin, Research Administrator 


Trial registration 


The protocol was published on the EEF website in December 2015 and no changes to the trial design 
were made after the trial start. The trial was registered on 6 July 2016 on the international standard 
randomised controlled trial number (ISRCTN) registry at htip://www.isrctn.com/ISRCTN64088368. 
Registration took place after the start of the trial but before any outcome data was received. 
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Methods 


Trial design 


The evaluation estimated the impact of the Switch-on intervention on reading outcomes using a three- 
arm school-randomised controlled effectiveness trial with waitlist. An effectiveness trial aims to assess 
outcomes under conditions as close as possible to those faced in the real world.' The trial’s primary 
focus was to compare the two treatment conditions jointly against a control condition, on the 
understanding that the differences between the interventions were marginal and of secondary interest. 


Switch-on was delivered from January 2016 to March 2016; 184 schools in 17 local areas were recruited 
and were randomly allocated to one of three groups: 


1. Switch-on Reading; 
2. Switch-on Reading and Writing; or 
3. a ‘business as usual’ control group, with a financial incentive. 


Schools in the control condition used their business as usual approach to supporting struggling readers. 
These schools were offered compensation of £1,500 upon completion of outcome testing, and were 
given the option to participate in Switch-on training in the following year (to be delivered to any year 
groups except the trial year group). The evaluation collected information on the nature of support 
provided in control schools. 


Before randomisation, each trial school designated between 4 and 16 eligible pupils from Year 3 to 
receive the intervention, subject to treatment allocation. All schools were informed that the outcomes of 
these core trial pupils would be assessed using an intention-to-treat approach (that only the outcomes 
of the pre-selected pupils would be considered in the impact analysis). The number of pre-selected core 
pupils was determined by the number of TAs each school assigned to the intervention. Each 
participating school had to commit between two and four TAs, identified prior to randomisation, and for 
each TA, schools selected between two and four pupils to receive the intervention. 


This design was chosen because it takes into account the resource limitations schools actually face 
when making choices about targeted interventions and so is appropriate for an effectiveness trial. The 
requirement for a minimum of two TAs per school was stipulated to ensure peer support was available 
and to avoid TAs working in isolation. 


Participant selection 


Schools 

Twenty-one Switch-on trainers recruited schools in 17 local areas.* The local areas were selected by 
the developers and, in many cases, were areas where they had existing professional links. Any state- 
maintained, academy, or private primary school located in one of the 17 participating areas where there 
was not already whole-school implementation of Reading Recovery (that is, where teachers and TAs 
had been trained in, and practiced, Reading Recovery principles)? were eligible to participate. 


' An efficacy trial, conversely, aims to detect the potential of an intervention under conditions as ideal as possible. 
? The participating local authorities were Birmingham, Bristol, Cambridgeshire, Cheshire, Coventry, Derbyshire, 
Northamptonshire, Kent and Medway, Lincolnshire, Nottingham City, Nottinghamshire, Stoke on Trent, Sandwell, 
Walsall, Warrington, Wiltshire, and Yorkshire. 

3 The rationale for this was that if there was already whole-school implementation of Reading Recovery, including 
among TAs, additional training in Switch-on would bring limited benefits. Nevertheless, the process evaluation 
indicated that 3% of intervention schools and 4% of control schools were delivering Reading Recovery. 
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Only TAs that had not already received training in Switch-on were eligible; there were no additional 
eligibility criteria. When a school agreed to participate in the trial, the headteacher was required to sign 
a memorandum of understanding setting out the roles and responsibilities of the school, the evaluators, 
and the Switch-on delivery team. The memorandum of understanding informed the school about the 
design of the trial, the consent procedures, and specified what type of data would be collected and 
when. 


Schools were only entered into the randomisation process if they completed a pre-randomisation survey 
and submitted all of the required data. Shortly after they were recruited to the trial, schools were sent a 
link to an online pre-intervention questionnaire that asked for information on: 


e type of school; 

e participating Year 3 pupils (such as their name, Unique Pupil Number (UPN), date of birth, and 
reading level);* 

e how TAs are trained and deployed in the school (for example, whether there was a standard 
induction for TAs, or whether time was timetabled for TAs and teachers to plan lessons or part 
of lessons together in advance); 

e TAs (including qualifications, experience levels in years, and which TAs would be working with 
pupils); 

e pupils selected to receive Switch-on and information on why they were selected; 

e ‘reserve’ pupils, in the event that those selected left the school before the start of the 
intervention; 

e rooms and equipment available for testing, and the dates of the summer term to plan the testing 
schedule; and 

e data on disability and SEN status in an anonymised, aggregated format. 


Pupils 

Pupils that had not met age-related expectations in literacy at the end of Key Stage 1 (based on teacher 
assessments carried out at the end of Year 2), and who did not have a high level of special needs, were 
eligible for selection to the core trial group. Schools were advised to only select pupils for whom, in their 
view, the intervention would be appropriate. If there were more eligible pupils than the maximum 16 
places available, schools were able to select which pupils to include. Pupils that were not pre-selected 
into the core trial group thus fell into two categories: eligible non-core pupils (those who were eligible 
but were not allocated to one of the places available) and ineligible pupils. These were included in the 
secondary analysis as set out below. 


When schools were recruited to the trial in September 2015, they were able to select three ‘reserve’ 
pupils who, if necessary, could replace pupils that left the school before the start of the treatment in 
January 2016. However, because of the potential for introducing bias in the group of core trial pupils, 
we did not consider any replacements in the analysis. 

Opt-out consent to participation in the trial, and to data linkage, was sought from parents of all pupils in 
Year 3 in all recruited schools prior to randomisation. The decision to obtain consent from all pupils was 
taken to allow (a) the investigation of spill-over effects on non-participating eligible and ineligible pupils 
and (b) whole-class outcome testing, which is less stigmatising for designated trial pupils and does not 
require opt-in consent. 


Outcomes measures 


The primary, and only, outcome measure used in the evaluation was the Hodder Group Reading Test 
(HGRT) 2A (Vincent and Crumpler, 2007).5 This test aims to test reading comprehension at word, 


4 Note that information on SEN was not collected at this point as this would have required opt-in consent from 
pupils and parents. School-level summary information was therefore collected instead. 
5 More detail is available at http://www.hoddereducation.co.uk/Product/9780340912706.aspx 
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sentence, and continuous text level and is suitable for use with mixed ability groups. It was chosen 
because it is targeted at a wide age range (Years 2-6, ages 7-12), thus reducing the risk of floor or 
ceiling effects, while being non-adaptive and therefore ensuring that all pupils are measured on the 
same dimensions. It is also paper-based, which allowed whole-class testing in primary schools without 
adequate IT equipment. HGRT 2 is highly reliable, with a Kuder-Richardson 20/Cronbach’s alpha score 
of 0.95. Improvement in writing skills from Switch-on Reading and Writing was not tested for as in this 
evaluation we were interested in the potential impact of the writing component on reading outcomes. 


Outcomes were collected through whole-class testing over a period of three and a half weeks between 
5 and 29 April 2016. Mop-up tests to collect outcomes from pupils that missed the main testing were 
carried out in 19 schools until 10 May 2016. Testing was overseen by NatCen invigilators who were 
blind to the treatment assignment of the school. Test papers were scored electronically by NatCen 
administrative staff who were also blind to treatment assignment. 


Sample size 


The study planned to recruit 189 schools, nine schools per Switch-on trainer. This meant that between 
756 and 3,024 pupils® were expected to be tested, depending on each school’s choices about the 
number of pre-selected pupils. For the power calculations, we assumed that the harmonic mean of 
pupils pre-selected in each school would be five,’ and the number of pupils in the trial therefore 945. 


Our initial power calculations were based on a joint comparison of Switch-on Reading and Switch-on 
Reading and Writing against the control. This is because our discussion with the developers and the 
EEF indicated that the two interventions were conceptually very similar and differences in outcomes of 
secondary importance. In addition to the overall question, each intervention was also tested separately 
against the control. Minimum detectable effect sizes (MDES), expressed in standard deviations, are 
displayed in Table 2. The calculations assumed a power of 80%, a statistical significance level of 5% 
for a one-sided test, and an intra-class correlation coefficient (ICC) of 0.15 (based on EEF 2015, which 
suggests an ICC of between 0.08 and 0.15 for Reading at Key Stage 2). 


The proportion of variance explained can be high for educational interventions if pre-test scores are 
used (Bloom et a/., 2007), and our primary MDES, based on an assumed sample size of 189 schools 
and a proportion of variance explained by pre-test scores of 0.60, was 0.18 standard deviations. 


These calculations did not account for attrition. 


Table 2: Minimal detectable effect sizes 


Unadjusted 0.22 
0.20* 0.21 
0.40* 0.19 
0.60* 0.18 


* Proportion of residual variance explained by baseline covariates. 


6 |f every school selected four pupils, 756 pupils would take part, if every school selected 16 pupils, 3,024 pupils 
would take part. 

7 The harmonic mean is the reciprocal of the arithmetic mean of the reciprocals, and, compared to the typical 
arithmetic mean, is less influenced by large outliers. 
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Randomisation 


Schools were randomly assigned to either of the two treatments or the control condition using 
randomisation stratified by the Switch-on trainer. As each trainer had capacity to support six schools in 
the implementation of Switch-on, each trainer was asked to recruit nine schools to the trial. In the event, 
six trainers recruited eight schools, 14 the expected nine, and one recruited ten schools—a total of 184 
schools. 


For the randomisation, a random sequence, generated by NatCen using Stata 12, was used to shuffle 
schools linked to each Switch-on trainer. The first three schools were assigned to Switch-on Reading, 
the next three to Switch-on Reading and Writing, and the remainder to the control. As a result, 63 
schools were assigned to each treatment and 58 to the control condition. This approach is a slight 
variation to the approach set out in the statistical analysis plan, in which a third of a trainer's schools 
would be allocated to each arm with a random tie-breaker if the number of schools was not divisible by 
three. The adopted approach ensures that each trainer supported the maximum number of schools in 
the trial, which was felt to be important in the context of an effectiveness trial. 


Analysis 


Baseline characteristics 
Baseline characteristics were summarised for core-trial pupils by intervention group. Variables available 
at pupil level were presented at pupil level, other variables were presented at school level. 


Continuous variables were summarised with descriptive statistics (n, mean, standard deviation, range 
and median). 


At school level, the comparison covered: 


e school type; 

e urban or rural status of schools; 

e English as additional language (EAL) status; and 
e Special Educational Needs (SEN) status. 


At the pupil level, the following baseline comparisons were presented: 


e Free School Meal (FSM) eligibility; 
e gender; 

e birth term; and 

e KS1 reading outcome. 


A summary of baseline characteristics is presented in the ‘Outcomes and Analysis’ section of the report. 


Primary analysis 

The primary analysis examined whether pupils participating in a Switch-on variant had better average 
reading outcomes four to five months after starting the programme than pupils receiving ‘business as 
usual’ support for their reading. 


Both treatment arms were compared jointly against the control using a multi-level model to account for 
heterogeneity between schools. This approach is in line with our original plan as set out in the trial 
protocol, but differs from the approach set out in the Statistical Analysis Plan (SAP). The reason for this 
is that an external review of the draft SAP suggested a separate analysis. This was then codified. 
However, a review of the logic behind our original choice and considerations of statistical power caused 
a reversal to our original plan. We have carried out the analysis set out in the SAP as sensitivity analysis, 
as described below. 
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Analysis was conducted on an intention-to-treat basis. The main model adjusted estimates and 
standard errors to baseline reading outcomes (Key Stage 1 teacher reading assessment) and the 
randomisation stratification variable, the Switch-on trainer, as fixed effects. Confidence intervals were 
calculated using bootstrapping. 


The calculation of the effect sizes followed the methodology for Hedges’ g—using the differences in 
means between treatment and control groups as numerator, and the pooled standard deviation based 
on the total variance from a multilevel model as denominator. The effect size calculation was based on 
Hedges’ (2007) approach assuming unequal sample size per school. 


The analysis was run in R version 3.2.1 (R Core Team, 2015) using the EEF’s analytical package for 
education trials, eefAnalytics, in version 1.04 (Kasim et al., 2016). Confidence intervals were specified 
at the 5% significance level for a two-sided test. 


Sensitivity analyses 
A range of sensitivity analyses were carried out to explore the robustness of the main finding: 


1. Each treatment arm was compared separately against the control, as set out in the SAP. The 
analysis assumed coefficients for the included covariates were constant across treatment 
conditions. No adjustment was made for multiple comparisons as the outcomes of the two 
interventions are considered to be highly correlated and findings in one intervention are likely 
to support the finding in the other (Schulz and Grimes, 2005). 

2. Awider range of prognostic covariates were included to increase power. These were, at school 
level: a binary urban/rural indicator for the school, the type of school,® the total number of pupils, 
the proportion of pupils eligible for FSM, and the proportion of pupils with English as additional 
language. At year-group level: the proportion of SEN pupils, and the average baseline reading 
score. At individual level: gender, absences in Year 2, FSM eligibility, age, and the school term 
of birth. 

3. CACE analysis: any core-trial participants that had not participated in at least 50% of sessions 
were labelled as non-compliant and a complier average treatment effect (CACE) was 
estimated. The cut-off was based on the consideration that attending less than 50% of sessions 
could not be considered as having participated in the intervention in a meaningful way. 

4. Imputed values: we explored whether loss-to-follow-up was related to existing covariates or 
treatment assignment, used multiple imputation to infer the likely results of those lost to follow- 
up, and included these imputed data in a sensitivity analysis. In addition, although not specified 
in the analysis plan, we carried out an analysis that did not assume missing outcome data was 
‘missing at random’ (MAR) but is instead ‘missing not at random’ (MNAR). For this purpose we 
assumed that pupils with missing outcome data would have achieved a score of zero points. 


Our analysis plan also specified a variant of the CACE analysis where compliance was defined as 
attending at least 50% of sessions and being taught by a TA who had received the Switch-on training. 
However, due to the absence of the relevant TA data we could not perform this analysis. 


Secondary analyses 
In secondary analyses, four hypotheses were examined, as set out below. As these analyses are 
additional to the main focus of the trial, the results should be considered preliminary and indicative. 


e H2: pupils participating in Switch-on Reading have different (better or worse) outcomes than 
pupils participating in Switch-on Reading and Writing. 

e H3: non-participating eligible pupils in the same class as pupils who participate in Switch-on 
Reading or Switch-on Reading and Writing will have better reading outcomes than eligible 
pupils in classes without pupils who participate in either Switch-on programme (positive spill- 


8 The school types included: LA maintained schools, academies, faith schools and special schools. 
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over to non-participating eligible pupils). Pupils in treated arms are considered separately, 
reflecting the idea that core pupils’ learning (which may be different in the two interventions) is 
passed on to non-participating pupils, for example by raising expectations. 

e H4: non-participating ineligible pupils in the same class as pupils who participate in Switch-on 
Reading or Switch-on Reading and Writing will have better reading outcomes than ineligible 
pupils in classes without pupils who participate in either Switch-on programme (positive spill- 
over to non-participating ineligible pupils). Treatment arms, again, are compared separately. 

e H5: Switch-on Reading and Switch-on Reading and Writing (jointly considered) will have a 
different (higher or lower) impact on FSM pupils. 


Analyses for H2 to H4 included the full set of predictive covariates set out in the first sensitivity analysis 
above. The analysis of H5 included the same set of covariates as the primary analysis, to allow for 
comparability of the results. 


The Statistical Analysis Plan is included in Appendix C. 
Implementation and process evaluation 


The process evaluation was designed to explore how school staff in participating schools (TAs, class 
teachers, and co-ordinators) and Switch-on trainers viewed the delivery and implementation of 
Switch-on, and what they perceived the outcomes to be. 


The main research questions were: 


e What are the key success factors and barriers to successful implementation in a scaled up 
‘train the trainer’ model? 


e To what extent is fidelity to the core elements of Switch-on maintained when implementation 
is scaled up? 


e What are the direct and indirect costs of delivering Switch-on? 


Fieldwork took place between February and March 2016. The components are shown in Table 3. 


Table 3: Elements of the process evaluation 


Elements of the process evaluation 


Observation of a Switch-on Reading training session | Face-to-face by NatCen researchers 


Registers to collect information on pupil attendance | Pupil attendance log kept by TAs and 


at Switch-on sessions provided to NatCen via Switch-on trainers 
Interviews with TAs Over the phone by NatCen researchers 
Interviews with Switch-on trainers Over the phone by NatCen researchers 


Joint interview with the developers Over the phone by NatCen researchers 


Observation and interviews in six case-study | Face-to-face by NatCen researchers 
schools 


Post-intervention survey Online 


All research participants were provided with written information about taking part in the evaluation. 
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Observation of a Switch-on training session 


Observation of a ‘train the trainer’ session and a TA training session provided the research team with 
insight into the delivery of the programme. Information gained from these events was used to develop 
the research tools for the process evaluation. 


Registers to collect information on pupil attendance at Switch-on session 


During the ten-week Switch-on intervention period, TAs in the treatment schools kept a record of pupil 
attendance, and noted any absences. They passed this information on to their Switch-on trainers, 
who entered it into a pro forma provided by the research team and uploaded it to a secure File 
Transfer Protocol (FTP) site so NatCen could access it. Trainers also provided information on 
attendance at the TA training sessions and support visits. This information on the ‘dosage’ of the 
intervention was analysed and informed an assessment of whether fidelity had been maintained in 
implementation. 


Teaching Assistants 


TAs from treatment schools were interviewed in order to understand their experience of delivering the 
programme, how Switch-on fitted into school life, and what they perceived the outcomes, if any, to be. 
An original sample of 14 TAs was selected to include a good range on the basis of the following 
characteristics: 


e intervention group (Switch-on Reading or Switch-on Reading and Writing); 

e number of years of experience; 

e school practices in relation to TA deployment (induction training and whether TAs have 
planning time before lessons); and 

e size of school. 


A total of eight TAs were unable to participate. In their place, TAs with similar profiles were 
approached. 


TAs in the sample received an email with an information leaflet and were given the opportunity to opt- 
out. 


The interviews collected the views of TAs on a range of topics including: 


e the training they received; 

e = their workload; 

e delivering the programme; 

e materials; 

e views on whether the programme works more successfully with some groups than others, and 
why; 

e views on the facilitators to successful implementation; 

e views on the barriers to successful implementation; 

e outcomes for children and reasons for this; and 

e suggestions for improvements to the programme. 


Switch-on trainers 


These interviews aimed to understand trainers’ experience of the training they had received from the 
developers, and their experience of subsequently delivering training to TAs. They also examined 
views on barriers to implementation and examples of successful implementation. Five trainers were 
selected to participate from different local authority areas in different regions. 
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Case studies 


The aim of the case studies was to build a more detailed understanding of implementation. Day visits 
were carried out to selected schools. Visits took place around the eighth and ninth weeks of the ten- 
week Switch-on programme and included: 


e watching a Switch-on session to view the programme in practice; and 
e interviews with a TA, classroom teacher, and school co-ordinator. 


A sample of six schools was selected, half of which had participated in Switch-on Reading, and half in 
Switch-on Reading and Writing. In addition, across the sample, there were schools with different: 


e proportions of FSM pupils (low/medium/high); 

e school practices in relation to TA deployment (induction training; whether TAs have planning 
time before lessons); 

e sizes; and 

e academy/LA status. 


Two of the selected schools were unable to participate. In their place, two schools with similar profiles 
were approached and recruited. 


The classroom teacher interviews covered topics such as: 


the workload created by the programme; 

if, and how, teachers had worked with Switch-on TAs; 

the barriers and facilitators to implementation; 

their views on the content of the programme and its suitability for their pupils; 
how the programme fitted in with their curriculum and schemes of work; 
perceptions of benefits to pupils; and 

suggestions for improvements to the programme. 


The school co-ordinator interviews covered similar topics to the classroom teacher interviews 
including: 


e the workload created by the programme; 

e the barriers and facilitators to implementation and to sustaining the programme across the ten 
weeks; and 

e any improvements they would make to the programme. 


Developers 


One joint interview was conducted with the two developers. This enabled the research team to fully 
understand the rationale behind the programme, its aims and objectives, how it was intended to be 
implemented, and their views on how implementation had gone in practice. 


Conduct of interviews 


Each interview was based on a topic guide to ensure systematic coverage of key issues that 
addressed the process evaluation research objectives. It was intended to be flexible and interactive, 
allowing issues of relevance to be covered through detailed follow-up questioning. Separate topic 
guides were produced for each type of respondent. 


To minimise the burden on participants, all interviews, apart from the case studies, were conducted 
over the telephone. Interviews lasted approximately 45 minutes. The interviews were digitally 
recorded and then analysed using Framework, a systematic approach to qualitative data 
management developed by NatCen Social Research and now widely used in social policy research. 
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All participants were assured that everything discussed in the interview would remain confidential and 
would be treated in accordance with the Data Protection Act. 


Post-intervention survey 


At the end of the intervention, intervention and control schools were sent a link to a short online post- 
intervention survey. Its purpose was to map intervention and delivery in implementation schools and 
to gather an overview of the other interventions that may have been delivered in control schools—to 
understand what ‘business as usual’ meant for each school. The link to the survey was sent to the 
Switch-on co-ordinator in each school. Dosage and fidelity information was collected by asking 
schools whether the same TAs worked with the pupils throughout the programme, whether any TAs 
left the school, whether any pupils left the school, and whether the programme was delivered fully in 
the way it was intended. The survey also collected cost information from all schools. 


Of the 184 participating schools, 179 completed post-intervention surveys—a response rate of 97%. 
Costs 
Information on costs was collected through two tools: 


e the post-intervention survey; and 
e acost spreadsheet, provided by the developers, covering all aspects of the delivery, discussed 
with them in a subsequent interview. 


Our estimate of the total cost includes up-front costs (such as training for TAs and the purchase of 
resources and materials) and running costs (such as TA time to prepare and deliver the Switch-on 
sessions, and photocopying costs). The costs of developing the intervention are not included. 


In order to calculate the total cost per pupil in the intervention’s first year we considered those pupils 
who were intended to receive either Switch-on Reading or Switch-on Reading and Writing (n = 658). 
When estimating costs per pupil in the intervention’s second and third years, we assumed that the same 
number of pupils would receive the intervention. 


Timeline 


Planning for the intervention began in Spring 2015 with initial discussions about the trial with schools. 
This was followed by memorandums of understanding being signed at the start of the Autumn 2015 
term. The intervention was delivered during the Spring 2016 term. Evaluation activities in the field 
started in January 2016 and ended in May 2016. Table 4 sets out the key evaluation milestones. 


Table 4: High level evaluation timeline 


March—September 2015 | Recruitment of schools 


Memorandum of understanding signed 
September 2015 
Opt-out letters sent to parents by schools 


Saabera0ié Schools provide pre-randomisation data on selected pupils 
ctober 
Randomisation of schools oa based on data received 


November—December 
2015 


January 2016 Intervention delivery begins 
February 2016 Fieldwork for the process evaluation 


TAs receive Tasrecevetrainng 
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March 2016 Intervention finishes 
April 2016 Pupil testing 


May 2016 Schools complete the online post-intervention survey 
August 2016 NPD outcome data extract obtained 
September 2016 Analysis 
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Impact evaluation 


Participants 


Switch-on trainers approached 476 schools across the 21 areas in which they operated,° and recruited 
184 schools from a target of 189. One of the recruited schools (the only special school) was excluded 
from the study after randomisation because teachers did not believe the outcome test was appropriate 
for their pupils.1° 


In the 184 schools, 9,034 out of 9,185 (98%) pupils provided consent to trial participation and data 
linkage. Among those providing consent, 2,064 pupils were eligible for the intervention and 999 were 
pre-selected by schools to participate in the core trial. Eligible pupils had to be in Year 3 (aged seven 
and eight years), working below age-related expectations at the end of Key Stage 1 (five to seven 
years), and without a high level of special needs. Schools were able to select pupils who met these 
criteria and were advised by the Switch-on trainers and evaluation team to only select pupils for whom, 
in their view, the intervention would be appropriate and beneficial. 


Of the 999 pre-selected pupils, 80 were lost to follow-up, so that outcome data of 919 pupils (92% of 
those assigned) was available for the analysis. For an additional 17 pupils, other covariates included in 
the primary analysis were missing, meaning that 902 pupils (90%) were included in the primary analysis. 


° Switch-on trainers were responsible for recruiting schools and used a variety of different techniques for that 
purpose. Some trainers invited all primary schools in their area, while others recruited in a more targeted way. As 
a result, the number of schools approached by trainers is not an appropriate indicator of schools’ interest in 
participating in the study. 

10 At recruitment, teachers—on the basis of the intervention description given by trainers—believed that their pupils 
could benefit from this intervention. For this reason, the special school was not excluded prior to randomisation. 
However, when schools were asked to plan the outcome testing phase, the special school teachers made it clear 
that the test would not be appropriate for their pupils. At this point the decision was made to drop the school from 
the study. 
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Figure 3: Participant flow diagram 
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Table 5 sets out the changes in the minimum detectable effect size (MDES) between design and 
analysis stages. At design stage, the MDES was estimated to be around 0.18 standard deviations, 
based on the assumptions that the two treatment arms would be jointly compared against the control in 
the main analysis, that baseline covariates would explain around 60% of the variance of final outcomes, 
and that the strength of the previous evidence allowed the significance test to be one-sided rather than 
two-sided. 


The MDES at randomisation stage increased to 0.24 standard deviations due to the following factors: 


e It was also decided to use, despite the positive evidence from the efficacy trial, a two-sided test 
to conservatively allow testing for negative intervention effects. This was done to better reflect 
current practice in impact evaluation research. 
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e The baseline covariates were not very predictive of the outcome test results. Our original power 
calculation assumed that baseline covariates would explain around 60% of variance in the 
outcome. However, the NPD-based baseline only accounted for 12% of variance (the 
correlation between baseline and outcome measure was 35%). 


It is difficult to know exactly why baseline measures were such a poor predictor of final results. The 
higher correlation for better-performing pupils (66% for pupils not eligible for the intervention) indicates 
that regression to the mean may be a factor, a phenomenon whereby extreme scores normalise towards 
the mean. Regression to the mean occurs due to a proportion of extreme values being the result of 
chance variation. Another reason may be the difficulty in assessing reading skills adequately at KS1, or 
the importance of factors not accounted for in the analysis in influencing reading skills during the time 
of the intervention. Finally, the HGRT outcome test may not be a reliable reading test for low-achieving 
‘eligible’ pupils, despite the overall positive evidence on reliability and validity from the standardisation 
across the range of skills, which showed high internal validity and good high correlation with other 
measures of reading ability (see Vincent and Crumpler, 2007). 


The MDES at analysis stage increased marginally to 0.25 due to loss-to-follow-up of some pupils. 


Table 5: Minimum detectable effect size at different stages 


Correlation 
N between Blocking/ Minimum 
[schools/pupils] | pre-test stratification fo (=) (Tey te) () 
(n=intervention; (+other or pair effect size 
aEexe) ai igo) )) covariates) matching (V2) 5) 
& post-test 
Schools: 189 
(SO: 126; C: 63) 
Protocol Pupils: 945 
(SO: 630; C: 315) 


Stratification 
by SO trainer 


Schools: 183 
(SO: 125; C: 58) 
Pupils: 902 

(SO: 606; C: 296) 


Analysis (i.e. 
available pre- 
and post-test) 


Stratification 
by SO trainer 


ce 
Schools: 184 
(SO: 126; C: 58) — 
Randomisation | Pupils: 999 ee oth 
(SO: 658; C: 341) y 


School and pupil characteristics 


Of the 184 schools recruited to the study, one, a special school, was excluded shortly after 
randomisation due to concerns of teachers about pupil testing. Of the remaining 183, seven in ten (69%) 
were local authority maintained schools, two in ten (22%) academies, and one in ten (9%) faith schools. 
Nearly nine in ten (87%) were located in an urban area. 


The cross-trial mean of the school-level proportions of pupils with English as additional language was 
16.5%, and of pupils with special educational needs, 11.5%. On average, a trial school had 50.1 Year 
3 pupils, of which 16.9 were eligible for the intervention. Each trial school, on average, assigned 2.4 
TAs to the intervention. None of these characteristics differ statistically significantly between trial arms. 
However, there is a difference, statistically significant at the 5% level, in the number of pupils pre- 
selected by schools for the intervention. Control schools selected 5.9 pupils, Switch-on Reading schools 
5.6, and Switch-on Reading and Writing, 4.9. 
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At pupil level, across schools, 40% of core trial pupils were FSM-eligible, and 42% of core trial pupils 
were female. Pupils missed around 5% of the lessons they could have attended. These percentages 
do not differ between treatment arms. This is set out in more detail in Table 6. 


Importantly, there is no statistically significant difference in baseline scores (KS1 reading score) across 
treatment arms, with 11.17 points achieved, on average, in Switch-on Reading and Writing as well as 
the control group, and 11.09 points achieved in Switch-on Reading. The effect size of the difference is 
0.00 for Switch-on Reading and Writing against the control, and 0.03 for Switch-on Reading against the 


control. 


Table 6: Baseline comparison 


Variable Switch-on Control 
School-level n/N (missing) Percentage n/N (missing) Percentage 
(oF (To fo) dler-1)) 

School-type 

Local authority 

maintained 89/125 (1) 71% 38/58 (0) 66% 
school 

Academy 25/125 (1) 20% 15/58 (0) 26% 
Faith school 11/125 (1) 9% 5/58 (0) 9% 
Rural/urban 

status 

Rural 13/125 (1) 10% 10/56 (2) 18% 
Urban 112/125 (1) 90% 46/56 (2) 82% 
foaiede n (missing) Mean n (missing) Mean 
EAL (% of pupils 

per school) 123 (8) 16.5 53 (5) 16.4 
SEN (% of pupils 

in Year 3) 125 (1) 11.7 58 (0) 11.3 
Average number 

of Y3 pupils per 125 (1) 50.8 58 (0) 48.7 
school 

Number of 

eligible Y3 125 (1) 16.7 58 (0) 17.4 
pupils 

Number of pre- 

selected (core) 125 (1) 5.2 58 (0) 5.9 
Y3 trial pupils 

TAs selected to 

deliver the 

intervention, per ten) ee 3340) ae 
school 

fen n/N (missing) Percentage n/N (missing) Percentage 
Eligible for FSM 221/606 (52) 36% 115/296 (45) 39% 
Gender (female) 256/606 (52) 42% 122/296 (45) 41% 
Birth term 

Spring 273/606 (52) 45% 129/296 (45) 44% 
Summer 239/606 (52) 39% 113/296 (45) 38% 
Autumn 94/606 (52) 16% 54/296 (45) 18% 
eontinsue) n (missing) Mean n (missing) Mean 
Koi reading 606 (52) 11.1 296 (45) 11.2 


score (baseline) 
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Absences 


(%) 606 (52) 4.9 296 (45) 5.0 


There is a Statistically significant difference in loss-to-follow-up between trial arms in the core trial group 
(see Table 7). The Switch-on Reading arm lost 7.4% of pupils at follow-up, Switch-on Reading and 
Writing 5.2%, and the control group 11.1%. Differential loss-to-follow-up can introduce bias in effect 
estimates to the extent that the differences are related to outcomes, that is, that the likelinood of being 
a poor or good reader is linked to the likelinood of missing the outcome test. 


The large and statistically significant difference in loss-to-follow-up between trial groups raises the 
possibility that loss-to-follow-up is linked to outcomes, since eligible non-participating pupils were 
substantially more likely to miss the outcome test (23% did so) than ineligible, adequate readers (of 
whom 5% missed the test). Teachers may have felt that participation in a test was not appropriate for 
some pupils who would have achieved lower scores. However, it may also reflect a higher likelihood of 
absences in the eligible compared to the ineligible group. 


The difference in missingness between trial arms of the core group thus may be problematic if it reflects 
teachers’ propensity to prevent poorly performing pupils from being tested, or less problematic if other, 
less strongly outcome-related reasons are the cause. The imputation of missing outcome values thus 
becomes an important sensitivity analysis. 


Table 7: Percentage of pupils lost to follow-up, per trial group and treatment arm 


Oxo) al ine) | SOR SO RW 


% %o %o 
Core 11.1 7.4 5.2 
Eligible 23.1 25.1 19.9 
Ineligible 5.0 5.7 4.2 


Outcomes and analysis 


Primary analysis 

The average post-intervention outcome scores for the core trial group, as measured by the Hodder 
Group Reading Test 2a, were 15.99 and 15.81 for Switch-on and the control group, respectively. The 
estimate of the average treatment effect, rounded to two decimal places, was 0.00. (Table 8). 


Table 8: Raw outcome test scores by treatment arm 


Pi celamel am atsr-(el are) fey alice) me] cell] e) 


Number of pupils 

Pate ae 606 (52) 296 (45) 
Mean 

(95% confidence 15.99 (15.54, 16.44) 15.81 (14.99, 16.63) 
interval) 
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Effect size of 
comparison against 0.00 
control group 


Confidence interval for 
comparison against (-0.13, 0.13) 
control 


Intra-class correlation 
in comparison against 0.16 
control 


Sensitivity analyses 
The three sets of sensitivity analyses support the robustness of the primary analysis estimates. 


1. Comparing Switch-on Reading and Switch-on Reading and Writing separately against the control 
Our Statistical Analysis Plan differed from our original protocol and the primary analysis presented by 
proposing to analyse Switch-on Reading and Switch-on Reading and Writing separately against the 
control. It was felt, at the time, that this was necessary to reflect the remaining uncertainty about the 
importance of the differences between the two intervention forms. However, following discussions with 
the EEF, we decided to revert to our original plan of a joint analysis in order to maximise statistical 
power. For transparency, we present here the analysis proposed in the SAP, which comes to the same 
conclusion of no evidence of an effect. 


2. Wider range of prognostic factors 
In the first sensitivity analysis we included additional prognostic factors to reduce the variance around 
the estimated effect and offset the influence of any baseline imbalances in these factors. We added: 


At school level: 


e =abinary urban/rural indicator for the school; 

e the type of school;"' 

e the total number of pupils; 

e the proportion of pupils eligible for FSM; and 

e the proportion of pupils with English as additional language. 


At year-group level: 


e the proportion of pupils with special educational needs; and 
e the average baseline reading score. 


At individual level: 


e gender; 

e absences in Year 2; 

e FSM eligibility; 

e age; and 

e the school term of birth. 


'! The school types included: LA maintained schools, academies, faith schools, and special schools. 
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The results, set out in Table 9, show no change in the estimated effect of 0.00. As before, this estimate 
is not statistically significant. 


3. Complier average causal effect 

Non-attendance, or, in the case of the control group, take-up of treatment despite control group 
assignment, can lead to an underestimation of the true treatment effect when using an intention-to-treat 
approach. The estimation of the complier average causal effect (CACE) aims to take account of non- 
compliance to offer a better estimate of the true treatment effect. A core assumption of CACE analysis 
is that treatment outcomes are not dependent on the random allocation, in other words, that non- 
compliers in treatment and control groups have the same expected outcome as that of compliers in the 
alternative group. 


In the present case, the difference of CACE estimates to the primary analysis estimates will be small 
because compliance was very high. No control-group pupils took up the treatment. In the treatment 
group, over seven in ten (71%) of all possible treatment sessions were attended by pupils across the 
two treatment arms, and nearly nine in ten (89%) of pupils attended at least 50% of sessions. 


The point estimate of the effect of Switch-on on pupils attending at least 50% of sessions is -0.01 (Cl: - 
0.16 to 0.14), and for pupils attending at least 80% of sessions -0.02 (CI: -0.29 to 0.25). 


4. Loss-to-follow-up 

Finally, we examined the effect of loss-to-follow-up on estimated effects under two different 
assumptions concerning missing data mechanisms: that outcome data were missing at random (MAR) 
and missing not at random (MNAR). MAR assumes that, except for random variation, missing 
outcomes can be predicted from other existing data, without bias. For example, pupils with higher levels 
of absence are more likely to perform poorly in the outcome test, but their performance does not also 
depend on, say, their teachers’ judgement on how well the pupil would do in the test (on which we have 
no data). MNAR assumes that the missing data is related to the values of unobserved data. For 
example, participants judged by teachers to have lower levels of literacy may be less likely to complete 
the outcome measure. It is important to note that we cannot know whether our missing data is missing 
at random or missing not at random. 


Assessment under the MAR assumption started, first, by exploring whether loss-to-follow-up was 
related to any prognostic variable or treatment assignment and, second, by running our expanded 
model with imputed missing outcome values. 


Loss-to-follow-up rates were 7.4% in the Switch-on Reading arm, 5.2% in the Switch-on Reading and 
Writing arm, and 11.1% in the control arm. 


A multi-level logit model showed that the likelihood of missing data was predicted by higher absences, 
lower treatment attendance rates in the treatment groups, attending a faith school, and the trial arm. 


Once imputed missing outcomes were included in the model, the estimated effect (0.03 standard 
deviations) was of similar magnitude as the effect estimated in the primary analysis, and not statistically 
significant. 


Under the MNAR assumption, we assumed that all pupils with missing outcomes would have achieved 
the lowest point score (zero points) if they had taken part in the outcome test. This is an extreme 
assumption that builds on the idea that missingness is related to teachers’ insight into pupils’ 
performance. In reality, even if teachers had held back low-performing pupils from the test, those pupils 
would have, on average, achieved scores higher than zero. Given the distribution of missing values, 
our assumption is likely to bias results in favour of finding an intervention effect. We found that under 
these assumptions, the estimated effects are larger (0.12 standard deviations) but not statistically 
significant, thus supporting our main result. 
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Table 9: Effect sizes of sensitivity analyses 


n (missing) Effect size Confidence intervals 


Separate analysis of Switch-on Reading and Switch-on Reading and Writing 


Switch-on Reading 612 (79) -0.04 (-0.19, 0.11) 
Switch-on Reading 

and Writing 586 (63) 0.05 (-0.10, 0.19) 
Additional prognostic factors 

Switch-on vs. control 849 (150) 0.00 (-0.15, 0.15) 


Complier Average Causal Effect (CACE) 


Attended at least 50% 


of sessions 860 (139) -0.01 (-0.16 0.14) 


Attended at least 80% 


of sessions 860 (139) -0.02 (-0.29, 0.25) 


Imputed outcome values (multiple imputation assuming MAR) 


Switch-on vs. control 978 (21) 0.03 (-0.12, 0.13) 


Imputed outcome values (manual imputation assuming MNAR) 


Switch-on vs. control 974 (25) 0.12 (-0.02, 0.23) 


Secondary analyses 
Four additional analyses were carried out, as specified in the analysis plan. The results are summarised 
in Table 10. 


H2: Difference in impact between Switch-on Reading and Switch-on Reading and Writing 

The first compared the two treatment arms, Switch-on Reading with Switch-on Reading and Writing. 
We found no difference in effect between the two treatment variants, although it should be noted that 
the trial was powered to detect only very large differences with certainty. The point estimate, not 
statistically significant, was 0.08 standard deviations (Cl: -0.05, 0.22) in favour of Switch-on Reading 
and Writing. 


H3: Impact on non-participating eligible pupils 

Our second analysis explored the effect of the Switch-on variants on pupils who were eligible to receive 
Switch-on but were not selected by schools to receive the intervention. We had hypothesised that any 
improvement in reading ability of core-trial pupils could raise expectations and thus standards of other 
eligible, as well as ineligible, pupils. However, our result—a moderate and statistically significant 
negative effect of 0.31 standard deviations—indicates that Switch-on could have a negative effect on 
non-participating eligible pupils. One could speculate that the reduction in availability of TAs to non- 
participating pupils with lower reading standards has a negative effect on their performance. However, 
it needs to be emphasised that these results, even if statistically significant, are only indicative. One 
reason for caution is the large proportion of missing data. Nearly a third of observations were excluded 
from analysis due to missing data. Moreover, our trial was designed to test our primary hypothesis. 
Undertaking multiple hypothesis tests increases the likelinood that the groups being compared will 
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appear to differ on at least one attribute due to random sampling error alone and may lead to false 
conclusions that an observed difference is ‘real’. 


H4: Impact on ineligible pupils 

Our trial also pointed towards a positive spill-over effect on non-eligible pupils. Results showed a low, 
statistically significant positive effect (of 0.11 standard deviations) on the reading outcomes of ineligible 
pupils. In the light of our previous findings, these results could be hypothesised to arise from the removal 
of under-performing pupils from the classroom, although again, these findings are only indicative and 
may be spurious. 


H5: Impact on core trial pupils eligible for free school meals 

Our final secondary analysis explored the effect of the Switch-on variants on FSM pupils. A subgroup 
analysis showed no effect of either treatment arm. The point estimates in each case are more extreme 
than those of the full sample, so that Switch-on Reading shows a larger negative effect on FSM pupils, 
and Switch-on Reading and Writing a larger positive one. An interaction test also failed to find a 
statistically significant effect. 


Table 10: Effect sizes of secondary analyses 


n Effect size efey ay icel=tarers) Intra-class 

(missing) (Hedge’s g) interval correlation 
H2: Comparing Switch-on 
Reading and Writing over Switch- 606 (10) 0.08 (-0.05, 0.22) 0.21 
on Reading 
H8: Non-participating eligible 719 (346) 0.31 (-0.47, -0.13) 0.12 
pupils 
H4: Ineligible pupils 6309 (661) 0.11 (0.06, 0.17) 0.12 
H5: FSM subgroup analysis 336 (61) 0.03 (-0.18, 0.20) 0.17 

Additional checks 


In addition to the above analyses, we examined the possibility of floor effects in the outcome measure. 
Floor effects occur when a non-trivial proportion of trial participants (in either treatment arm) achieve 
the lowest possible score. In that case it is no longer possible to distinguish between better- and worse- 
performing pupils achieving this score, and consequently improvements due to the intervention may go 
undetected. 


There is no evidence of floor effects, as 0.3% of core trial pupils in the control group and 0.5% in the 
treatment group obtained the lowest possible score (zero). The histogram gives further assurance that 
floor effects were not a problem preventing the detection of an effect. 
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Histogram of core trial group 
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Figure 4: Histogram of trial outcomes for core group 


Cost 


The cost estimate presented here relates to the cost to schools of implementing the intervention without 
EEF funding. Our estimate of the cost of a school participating includes: 


e up-front costs, such as buying the Switch-on set of reading books, the Switch-on manual and 
teaching materials, fees for the initial and follow-up training, and staff time required for 
attendance at training; and 

e running costs, such as the time of TAs to prepare for and deliver sessions, the time of co- 
ordinators in schools to support TAs, and the costs of photocopying materials. 


Our estimates are based on 658 pupils receiving the intervention in 126 participating schools (63 in 
each treated arm) with 296 TAs and 126 co-ordinators. This equates to approximately five pupils, two 
or three TAs, and one co-ordinator per school. Costs incurred by the Switch-on developers to train the 
trainers and pay their professional fees are not included in our estimates. 


The following sections show the components of costs incurred by schools and present the cost per 
pupil. 


Overall, the cost of the intervention was estimated at £359,499, and the total cost per pupil, over the 
first year, was estimated at £546.35. 


Up-front costs and running costs 


The main up-front costs to schools are the one-off purchases of two sets of books (£1,850), the 
Switch-on manual (£30) and accompanying teaching materials (approximately £9 per TA), and fees 
for training. For Switch-on Reading and Writing, TAs and co-ordinators received two days (14 hours) 
of training, and for Switch-on Reading, one day (seven hours). In addition, TAs in each version of the 
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programme received half a day of follow-up training after they had started delivering the intervention. 
The fee for two days (fourteen hours) of training is approximately £300.12 


Delivering Switch-on required a considerable investment of time by TAs. As well as the daily 20- 
minute session with each pupil, schools reported that each TA spent an average of 1.5 hours per day 
preparing to deliver Switch-on and writing up notes after sessions. Schools also reported that co- 
ordinators spent around 1.5 days (10.5 hours) across the ten-week period supporting TAs in their 
delivery. Therefore, on the basis of seven working hours per day, a total of 4,067 hours were spent by 
staff (co-ordinators and TAs) when preparing the sessions. Additionally, 33,390 hours were spent in 
total by staff when delivering the intervention. 


The only direct running costs identified relate to photocopying forms and ‘running records’, and the 
occasional purchase of additional books. This averaged approximately £13.75 per school (£1,733 in 
total). 


Table 11: Trial total up-front and running costs 


Time in Days 
Item (hours) 


Switch-on folder and teaching materials 


Time for co-ordinator to attend training (Reading and Writing) 126 (882) 
Time for co-ordinator to attend training ( Reading) 63 (441) 
Time for TAs to attend training (Reading and Writing) 370 (2,310) 
Time for TAs to attend training ( Reading) 222 (1,554) 
Books 233,100 
Fees for training (Reading and Writing) 74,400 


Fees for training (Reading) 42,750 


Travel costs to attend training 1,072 
Sub-Total 781 (5,467) 357,766 


Total up-front costs 357,766 


Time in Days 


Item (Hours) Cost (£) 
Time spent by TAs preparing sessions 1,586 (11,102) 
Time spent by TAs delivering sessions 1,410 (9,870) 
Time spent by TAs writing up sessions and follow up after sessions 1,586 (11,102) 
Time spent by co-ordinators supporting TAs 189 (1,323) 
Printing and photocopying/purchase of additional books 1,733 
Sub-Total 4,770 (33,390) 1,733 


12 It was reported that in the trial the cost of training—including venue hire, refreshments and TA travel but excluding 
trainer fees—was £90 for two days. It was also estimated by the developers that in a non-trial context and without 
EEF funding, the fees charged to schools for training would be £300 for two days (£150 per day). This is the figure 
that we have used in our estimates in order to reflect the true costs schools would incur if choosing to deliver 
Switch-on. 
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Total running costs | | 1,733 
Total: up-front and running costs 359,499 


Pupil cost and cost over time 


The up-front costs are regarded as one-off expenditure which will not apply in subsequent years if the 
intervention is extended for a longer period of time. This separation is particularly important as it also 
allows us to estimate the potential cost of the intervention if it is implemented for three years, where 
up-front costs are distributed over the whole length of the programme. 


Table 12 presents the up-front and running costs per pupil in the first year and Table 13 goes on to 
show how the average cost per pupil per year decreases as the length of the interventions is 
increased. 


The up-front cost per pupil for the trial is £543.72 and the running costs per pupil are £2.63 (although 
there is a significant investment of staff time) which gives a total per pupil of £546.35. In the case that 
this project is run over a period of three years, the per pupil per year cost decreases to £183.87. This 
is due to the up-front costs being relatively high compared to the total running costs. In terms of staff 
time required to deliver Switch-on, 8.3 hours per pupil were spent on TA and coordinator training and 
an additional 50.7 hours per pupil were required to deliver the sessions. This gives a total of 68.41 hours 
per pupil (or 8.44 days per pupil). 


Table 12: Up-front and running costs per pupil 


Cost Time in days (hours) per pupil Cost per pupil (£) 

Start-up cost 1.19 (8.31) 543.72 
Running costs per year | 7.24 (50.74) 2.63 
Total cost in first year 8.44 (68.41) 546.35 


Table 13: Costs per pupil over time 


When delivered for 1 year 546.35 546.35 


548.98 274.49 


When delivered for 3 years 551.62 183.87 


Per pupil costs compared with the efficacy trial 


The per pupil costs estimated in the previous efficacy trial was £627, which is higher than the £546 
per pupil estimated in this effectiveness trial. The cost of resources was similar in both trials (£1840 in 
the efficacy trial and £1850 in this study). The estimated per day cost of training was higher for this 
study than in the efficacy trial (£150 per day compared with £95 per day in the efficacy trial). 
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Process evaluation 


This section synthesises the findings from the process evaluation of the implementation of Switch-on. 
It sets out what worked well and the main challenges, and identifies areas of improvement that could 
strengthen fidelity and replication. It brings together the perspectives and experiences of stakeholders 
who participated in the evaluation research. These include Switch-on trainers who trained Teachings 
Assistants and supported intervention delivery, TAs who delivered the daily sessions for pupils, co- 
ordinators responsible for intervention delivery in their school, classroom teachers, and the developers 
of the intervention. Table 14 sets out the number of achieved interviews for each type of participant. 


Table 14: Achieved interviews, process evaluation 


Switch-on trainers 5 
Teaching Assistants 20 
Class teachers 6 
Co-ordinators 6 
Switch-on Developers 1 joint interview with both developers 


The process evaluation also drew on an online post-intervention survey: 179 out of the 184 participating 
schools completed the survey (a response rate of 97%). 


Implementation 


The section on implementation is structured around the key elements that were found to be related to 
successful delivery of Switch-on: 


attractiveness of the intervention; 

quality of the training and resources; 

team delivery; 

senior leadership, particularly in relation to resource management; 
halfway visit by the trainer to the school; 

adequate facilities; and 

‘Reading Recovery’ experience. 


Attractiveness of the intervention 


The intervention was attractive to stakeholders and aligned with the ambition of most schools to 
ensure pupils were confident and effective readers, and that TAs were skilled at supporting reading. 
TAs and co-ordinators were particularly enthusiastic about the new books. They considered that the 
books covered a good range of topics and fostered the varied interests of different pupils. 


‘We loved the fact that we were given all of these lovely books and everything and the 
children were going to get to read lots of new stories and everything because once you can 
get children enthusiastic about reading you're halfway there really (TA). 


The books were primarily narrative fiction. There was a clear rationale that non-fiction texts are less 
helpful for struggling readers due to their less ‘natural’ language, specific vocabulary, and more 
complicated layouts. Nonetheless, some TAs felt that the range of books could have been broader to 
introduce children to different reading experiences, for example by including non-fiction books. This 
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suggests that a clearer explanation of why fictional texts had been selected could be helpful to secure 
commitment among TAs. 


Another aspect of the perceived attractiveness of Switch-on was the upskilling of TAs. Stakeholders 
felt that Switch-on invested in TAs, increasing their understanding of reading support strategies and 
teaching them how to adapt their teaching to children’s individual needs. As a consequence, it was 
seen to enable TAs to take more responsibility in the classroom, which was highly valued. 


Quality of the training and resources 
The following aspects of the training facilitated effective implementation of Switch-on: 


e Explanations and examples were clear. TAs particularly praised the use of videos to 
demonstrate aspects of the intervention. 

e ~The training was practical and hands-on. For example, the role-play and mock delivery were 
valued by TAs. 

e Folders with all the forms and documents needed for the intervention were provided and 
these were considered to save TAs’ time during the intervention. 

e There was opportunity for school co-ordinators to attend as well as TAs. This meant co- 
ordinators had a good understanding of the intervention and of how they should support TAs. 
As aresult, TAs felt well-supported and were confident that the co-ordinator could help them 
to deliver the intervention. 

‘When | left the training, there was nothing that | needed that | hadn't got’ (TA). 

Similarly, TAs and co-ordinators generally felt well-supported by the trainer. 


Although the training videos were seen as useful, TAs felt they would have benefitted from seeing a 
video of a full session rather than just components of a session. This was provided later at the half- 
way training and also made available on the website, but earlier availability was thought to be 
important. Some TAs were not aware the training video was on the website. 


Reducing the lag between the end of training and the start of delivery would also support improved 
delivery. TAs felt less confident about implementing Switch-on because of the lag and had forgotten 
some aspects of the intervention, such as use of the Running Records. 


Ensuring materials are provided to schools well in advance of the start of intervention delivery was 
considered to be important for effective delivery. School staff found the resources (such as the books, 
notebooks and pens) helpful but suggested they should be provided earlier to allow sufficient time for 
preparation. 


Team delivery 


The Switch-on delivery model included support for schools from a Switch-on trainer and support for 
TAs from a school co-ordinator and at least one other TA in the school (Figure 5). 


Figure 5: Support built in to the Switch-on delivery model 
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These different levels of support and collaboration between trainers, co-ordinators and TAs were 
critical to the smooth running of Switch-on: 


e The trainer was an important source of support for the co-ordinator. Co-ordinators 
appreciated the trainers’ availability to answer questions about the administrative aspects of 
the intervention as well as more technical questions related to reading strategies and pupil 
support. This provided reassurance and helped to build confidence that they were delivering 
the intervention as intended. 

e TAs benefitted from having a dedicated co-ordinator in the school. A co-ordinator without other 
competing responsibilities, such as teaching, was thought to provide the ideal level of support. 


e The fact that there was a minimum of two Switch-on TAs in the school meant that staff could 
discuss the intervention together, trial it, and answer each other’s queries. This motivated TAs 
and helped sustain their engagement with the intervention. 


It was felt that class teachers’ input was missing from the delivery model and there were concerns 
about a lack of coherence between Switch-on sessions and classroom teaching. Class teachers had 
no formal role in delivery, and in some cases it was reported that they knew little about the 
intervention. It was felt that their involvement would have helped TAs: 


e to understand better the background and difficulties of intervention pupils; 

e to develop better strategies to work with these pupils; and 

e to establish coherence between the intervention and the classroom to create a transfer of 
skills. 


Class teachers not being involved in the timetabling process, and the intervention taking place during 
key lessons (English, maths, and science), was felt to be a potential source of concern and tensions. 
Teaching staff worried this could have a negative effect on pupils’ confidence and attainment. They 
were particularly concerned when pupils missed English classes and were worried that already- 
struggling pupils might fall further behind. Schools in which co-ordinators, TAs and class teachers had 
worked collaboratively and discussed the delivery and goals of the intervention tended to be more 
satisfied. They recommended discussing the objectives for the selected pupils and involving teaching 
staff in the timetabling of the intervention. 


Leadership from senior staff 


The memorandum of understanding signed by schools specified that a member of the senior 
management team should be responsible for Switch-on in the school. While in many schools a senior 
member of staff was allocated (for example, the deputy head or even headteacher), in some schools 
the Switch-on co-ordinator was a Year 3 or 4 teacher. 


Findings from the qualitative interviews indicate that when involved, senior staff were able to liberate 
TAs from their other duties, organise cover, and ring-fence TA time for the intervention. This was seen 
as a crucial condition of successful delivery and where this did not happen, it was challenging to 
deliver the intervention effectively. 


‘So all the practicalities of time, of resources, all that organisational side, is huge with any 
intervention, and it can make or break the pupil outcomes for that intervention. So however 
good your training is, your Switch-on training, if, if the TAs go back to school, and there isn't 
that will to release them from the classroom, for example, or if schools are not prepared to 
buy the reading resources, then that intervention fails’ (Trainer). 


Managing TA workloads 


There was a perception that the introduction of Switch-on to a school contributed to resource 
constraints and limited TA availability in other areas. Managing this effectively was seen as an 
important factor in successful delivery. 
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Due to their increased workload resulting from Switch-on, some TAs had to stop delivering other 
interventions and programmes. There was a reported concern that non-intervention children were 
missing out on support as a consequence. For example, one TA used to deliver the Daily Readers 
literacy intervention but had to stop when she started working on Switch-on. She was concerned that 
other pupils were falling behind. 


‘There's other children now that are not getting my time because this is taking my time’ (TA). 


TAs’ increased workloads as a result of Switch-on also meant that in some schools there was less TA 
support available in the classroom. Some class teachers reported that this was challenging, and TAs 
also felt that they were not fulfilling all their responsibilities since the start of Switch-on, for example, in 
relation to supporting behaviour management in the classroom. In some instances, other TAs, or the 
class teacher, had to take on the responsibilities of the Switch-on TA. 


Overall, 79% of school co-ordinators agreed or strongly agreed that because of the TAs’ involvement 
in Switch-on, other pupils received less support than they would have otherwise. 


Half-way visit by the trainer to the school 


The half-way visit by the trainer was identified as crucial. This opportunity to clarify uncertainties or 
ask for support and strategies was highly valued by TAs. As a result of the visit, in which the trainer 
observed the TA delivering a session and provided feedback, TAs felt more confident that they could 
deliver the intervention as intended. 


‘That was really good ’cause it made me more confident to know that | was doing the right 
thing, and she gave us quite a bit of feedback and some good tips on how to progress the 
children’ (TA). 


However, there was a concern about the timing of the visit. It was reported that the visit took place too 
late in the intervention, and as a consequence limited the capacity of TAs to correct their mistakes or 
address their difficulties. Respondents felt the visit should have happened during the first few weeks 
rather than half-way through. The same feedback was collected about the half-day training, which 
took place mid-way through intervention delivery. 


‘If you've made a mistake really, by then, you're making a mistake all the way through (TA). 
Adequate facilities 


Having an adequate room, ideally in a private setting, was identified as important for the delivery of 
the intervention given the emphasis on one-to-one attention for the pupil. Around a third of schools 
(34%) reported that it had been a challenge to find adequate facilities. For example, one school 
reported delivering the intervention in a corridor, which meant ongoing disruptions that inevitably 
affected the quality of delivery. 


As aconsequence of the lack of a designated and stable location, some TAs had to carry the 
materials between locations and set up the intervention every day. This became an additional source 
of pressure on their time. There was also some concern that the lack of consistency disrupted pupils. 
In cases where the intervention was scheduled at a regular and specific time of the day, it was 
generally easier to secure a stable room. 


Level of detail of the intervention 


Switch-on was reported to be a demanding intervention that required a lot of investment from TAs in 
terms of preparation, delivery time, and paperwork. Some schools felt that the requirements of the 
intervention, in terms of time required for preparation and planning, could have been clearer. 
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The rationale behind the paperwork was to help the TA develop a theory of how the pupil was 
learning in order to inform planning and teaching. For that purpose, TAs had to complete a form for 
each session detailing the familiar reading to be used, the reading objective, the strategies planned 
for praise and prompts, and how well these worked. In addition, TAs had to do the Running Record, 
which listed the number of errors made and calculated the error rate in order to assess the adequacy 
of the reading level. 


Although ‘Running Records’ were seen as useful to monitor progress— You can see those little things 
that you would never pick up on just listening (TA)—TAs felt that the forms for session planning were 
repetitive and overly detailed. This put pressure on their time, which they thought would have been 
better used with the children. Sixty-eight percent of schools reported that doing all the paperwork 
relating to the sessions had been a challenge and there was a general view that the paperwork could 
be simplified. 


Concern about the approach to books 


There was a shared concern about the requirement to read three or four books per session. The 
rationale for this was to introduce children to a wider range of books. However, it was feared to be 
disheartening and demotivating for children if they did not have the opportunity to complete a book. TAs 
felt that reading a book all the way through and getting to the end was part of the satisfaction of reading. 
As a result TAs feared that not being able to do so might be demotivating for the intervention pupils. 


‘That is one of the drawbacks that all of us found, that the children, especially early on, they 
get so caught up in the fact that now they can read they really want to read to the end of the 
book and Switch-on doesn't—it almost doesn't want you to read to the end of the book (TA). 


Training for Switch-on trainers emphasised that children should have access to the books outside of 
the Switch-on session so that they could complete the books, but this was not always clearly understood 
by TAs. The need to cover three or four texts also meant it was a challenge for TAs to keep to the 
timeframe of 20 minutes per session. 


Reading Recovery experience in the school 


Schools that had a member of staff who was trained in Reading Recovery appeared to have fewer 
difficulties in delivering Switch-on effectively. The methodology and monitoring requirements were 
perceived to be similar and expertise could be transferred to Switch-on. This was considered to be 
particularly helpful early on, when TAs were still getting used to the intervention. 


Fidelity 


Commitment to deliver Switch-on as intended was evident. There was a high level of fidelity in terms of 
delivering sessions at the prescribed frequency and ensuring trained TAs continued to run the sessions. 
Nonetheless, school staff made contextual adaptations for various reasons—primarily in relation to the 
format and content of sessions. These findings reflect those of the previous efficacy trial, which also 
found that there was a tendency for some staff to stray away from the explicit schedule. 


Regularity: Most schools delivered Switch-on daily, as required, but around one in ten schools (9%), 
however, reported that they were unable to deliver daily sessions. The process evaluation indicates that 
this was principally due to competing pressures on TAs’ time. For example, TAs were required to 
provide cover when class teachers were unavailable, support pupil assessment, or help with mock 
SATs. Events such as school trips were also reported to disrupt the daily timetable. In addition, pupil 
absence due to illness or medical appointments was a factor. 


Format: TAs adapted the format of Switch-on based on what they thought would work best for the 
pupils. For example, they reported letting the sessions run for longer than 20 minutes when they felt the 
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children would benefit from not being interrupted. In some instances sessions also took longer due to 
form-filling. 


‘It's like you almost need a stopwatch in front of you and that's—! didn't want to do that 
because you don't want to be clock-watching while you're trying to work with a child because 
it just interrupts the flow a bit really (TA). 


In some cases only one familiar reading was done instead of two. 


Session lead: |In a very small proportion of schools (5%), in case of temporary absence, the TAs 
allocated to the intervention were replaced by a TA who had not been trained by a Switch-on trainer. In 
such cases, the process evaluation indicates that the TA tended to explain the intervention to the 
replacement staff in order to maintain coherence in the delivery. 


Content: Content was sometimes adapted to children’s specific needs. For example, some children 
who had behavioural difficulties were not always asked to do the writing, or the writing component was 
left out entirely. In one case, a child with behavioural difficulties was taken out of the intervention 
completely as staff felt it was not appropriate for him. The main change to content was an added focus 
on comprehension as TAs felt there was not enough focus and time dedicated to it. Adaptations in this 
case included additional prompts on comprehension and vocabulary checking. 


‘I felt as though Switch-on doesn't give you enough time to focus on comprehension, and | 
feel sometimes that children can accelerate in their reading but not always totally understand 
what they are reading’ (TA). 


It was thought that, although children were improving in decoding words, their understanding didn’t 
always improve equally. Some TAs reported moving up book levels very fast with their pupils, but then 
having to go down because of insufficient comprehension. This raised uncertainties and meant TAs 
became unsure about when to move a child to the next reading level. 


Outcomes 

A. Perceived benefits of the intervention 

The perceived benefits were: 

increased confidence; 

increased motivation and interest in reading; 


increased skills for pupils; and 
positive effects for TAs. 


Increased confidence 


Increased confidence was widely reported as an outcome of Switch-on. TAs reported that pupils were 
more confident looking at stories and talking about them. They felt pupils were more willing to talk in 
class, and less apprehensive about reading and writing. This was partly attributed to the fact that the 
approach of Switch-on is to allow mistakes and accept them as a normal part of learning. Reducing the 
fear of failure was seen as important, especially for under-achieving and less confident children. 
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The one-to-one support was considered to be an important contributing factor to increased confidence. 
There was a view that having the full attention of an adult, interested in what they were saying, helped 
to build confidence and therefore achievement. 


‘| think as part of Switch-on you have to have conversations, so the fact that somebody is 
there interested in what she’s saying | think has made her come out of her shell a bit and she 
seems more chatty in class as well’ (TA). 


Increased interest in reading 


TAs reported that Switch-on increased interest in reading and that children were more interested in 
taking books home, which for some had not happened before. They would be curious about starting a 
new book in a session, and be more engaged reading it—asking questions and talking about it. This 
interest was often reported as being sustained in the classroom. TAs also reported that pupils would be 
motivated to progress in the book bands and colours. 


‘She’s going into class and she’s actually trying with the reading now where she was a bit 
unsure before with whether she was doing it right or not and understanding It’ (TA). 


Increased reading and writing skills for pupils 


Switch-on was believed to have increased the reading and writing skills of pupils. In terms of reading, 
it was perceived that pupils could break down words and decode better, leading to more coherent 
reading. In terms of writing, pupils were reported to be thinking about structure more, and in some cases 
structuring sentences more effectively. In general, TAs reported that children were learning to self- 
correct more as well as developing their own decoding strategies. 


Some TAs also thought that some pupils were applying what they had learnt in Switch-on sessions to 
their wider work in the classroom. However, class teachers tended to be more sceptical about progress. 
Some class teachers reported seeing progress in the decoding of words, but not in reading 
comprehension. 


Positive effects for TAs 


Class teachers and co-ordinators also reported that a benefit of the intervention was to be upskilling 
TAs. It was perceived to give them the confidence to make professional decisions, notably about when 
to move a pupil to the next reading level. 


‘So | think that transference of ownership, and also that, that confidence to make those 
professional decisions as being one of the things that is developing through, through the 
programme?’ (co-ordinator). 


TAs themselves thought that the training had been a good learning experience and useful for them. 
Less experienced TAs specifically reported that it had reinforced their understanding of reading 
strategies and how to support struggling pupils. Some commented that even if they did not have the 
opportunity to deliver Switch-on in the future, they would maintain a similar approach to reading support. 


B. Unintended or negative consequences 


39 


Switch-on Effectiveness Trial 


Disruption to classroom learning 


Co-ordinators, class teachers, and TAs reported concerns that Switch-on could be disruptive to 
classroom learning when pupils were taken out of the classroom during core topics (English, maths, 
and science). They worried that by missing key subjects they would experience delays in their learning 
and struggle to keep up with the curriculum and its planned progression. 


‘It means that they are missing whatever continuity we've planned and the progression we've 
planned throughout the week. Say one of them might have been doing their reading 
intervention today in maths, so they are now not as able to do tomorrow's maths lesson, 
because they've missed out a step in the progression that we've planned throughout the 
week.” (class teacher). 


Teaching staff were concerned that this could have a negative impact on pupil confidence and 
attainment. 


In cases where pupils were missing English classes, it was feared that the intervention might be 
hampering rather than supporting progress. 


‘So we write on a Monday morning for a longer period of time, and she’s missed those. That 
was something she often found quite tricky, writing for a long period, and she would be quite 
slow. So | think she’s going to struggle to get back some of those skills.’ 


Another concern was that when children didn’t miss the morning core lessons, they would then miss 
physical education or creative classes. It was felt that missing such classes could be detrimental and 
lead to decreased enjoyment and motivation at school. 


This approach of schools deciding how to schedule Switch-on differs slightly from the approach in the 
previous efficacy trial. In the efficacy trial, each pupil was given a schedule in which to come out of one 
normal class per day for 20 minutes at a time for the Switch-on session. The schedule was arranged 
so that parts of different lessons were missed. Despite this, evidence from the process evaluation of 
the efficacy trial indicated that schools strayed from the schedule due to the constraints of the secondary 
school timetable, resulting in some cases in similar challenges to those described here. 


Reduced support for non-intervention pupils 


Seventy-nine per cent of schools agreed or strongly agreed that because of the TAs’ involvement in 
Switch-on, other pupils received less support than they would have otherwise. The increase in the TAs’ 
workload meant that some had to stop delivering other interventions. There was a reported concern 
that children who were not receiving Switch-on were missing out on other interventions that were 
previously delivered. For example, one TA used to deliver Daily Readers but had to stop to deliver 
Switch-on. This TA was concerned that pupils were falling behind. In another school, a maths 
intervention had to be cancelled. 


‘There are other children now that are not getting my time because this is taking my time’ 
(TA). 


Pressure on other staff workload 
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TAs’ reported reduced availability to support class teachers with classroom preparation and activities. 
In some cases it was reported to be a source of tension. TAs also reported feelings of guilt and 
frustration for having less time for other responsibilities. 


‘Sometimes you feel like you're walking out on a sinking ship’ (TA). 


It was reported that, in some cases, other TAs or the class teacher had to take on the workload originally 
meant for the Switch-on TA. 


Formative findings 


The findings from the process evaluation suggest ways in which a scaled-up delivery of Switch-on could 
be improved. Some of these stem from the experiences and suggestions of trainers and school staff, 
and it is important to note that participants in an intervention, even when they are professionals, can 
have mistaken views about what works and what does not. 


Closer monitoring of delivery of the intervention. Although most schools complied with the required 
‘dosage’ (the delivery of daily sessions), there is evidence to suggest that fidelity was affected in some 
cases by modifications to the content, the duration of sessions, and delivery format. Closer monitoring 
of delivery, especially in the early stages of the 10-week delivery period, could help to ensure greater 
fidelity. 


Reducing the lag between the end of training and the start of delivery. Schools, particularly those without 
previous experience of Reading Recovery, found the intervention complex and reported that it took time 
to become familiar with the requirements. Reducing the time between the end of training and the start 
of delivery should be considered to allow TAs to apply learning as soon as possible. 


Specifying the role of the classroom teacher in the intervention should be considered in further 
development work. It was reported that in some schools class teachers had limited awareness of 
Switch-on. Being clear about the requirement for Switch-on to be embedded in a broader literacy 
strategy led by the class teacher could help to enhance outcomes. As part of this, specifying that class 
teachers should be involved in developing each pupil’s Switch-on timetable—deciding when Switch-on 
sessions should take place—should be considered. 


Ensuring the Switch-on co-ordinator is a member of the senior management team. This was specified 
as a requirement in the memorandum of understanding, but in practice some Switch-on co-ordinators 
were Year 3 or 4 teachers. Ensuring a senior member of staff has oversight of Switch-on would help to 
alleviate challenges around timetabling and resourcing. 


Number of books per session. TAs found it a rush to cover three or four books per session and some 
believed that pupils would have been more motivated if they had fewer texts. 


Use of the Running Records. Feedback from some TAs suggested that the Running Records and 
session records were not helpful. They claimed the forms were repetitive and did not facilitate a fuller 
understanding of pupils’ progress. 


Control group activity 


The control condition was ‘business-as-usual’, however there was some indication that control 
schools compensated for their treatment status assignment by increasing the amount of literacy 
interventions they delivered. Control and treatment schools were asked questions in a post- 
intervention survey about literacy interventions delivered in the Spring Term of 2016 (January-March), 
when Switch-on was being delivered. As set out in Table 15, the survey indicated that: 
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e 25% of the control schools indicated they did not have other literacy interventions in place, 
compared to 46% of the treatment schools; 

e 75% of the control schools had some kind of other literacy intervention; and 

e 4% of the control group schools used Reading Recovery. 


Table 15 summarises the findings. 


Table 15: Reading interventions implemented in trial schools during the trial period with Year 3 
pupils (multiple selections possible) 


Inference Training 
Nessy 


Other/(s) 
Don’t know 
n=179 
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Conclusion 


Ceo) aeq UY (e) ak 


1. Participating children in schools delivering either version of Switch-on made no additional 
progress in reading compared to similarly struggling children in ‘business as usual’ control 
schools. The 4 padlock security rating means that we have high confidence that there was no 
difference, and that this was due to Switch-on and not affected by other factors. 


2. A similar result was observed for children eligible for Free School Meals. 


3. The secondary analysis suggests that Switch-on might have affected children who did not 
actually receive the intervention by changing the make-up of their class or the capacity of their 
TAs. These findings are tentative, but emphasise the importance of considering potential 
impacts on all children when using targeted interventions. 


4. Overall, participating staff were positive about the intervention and accompanying training. 
Perceived outcomes for pupils included increased confidence, motivation and interest in 
reading, and improved reading and writing skills. Skills of participating TAs were also felt to have 
improved. 


5. Some schools reported modifications to the prescribed content, duration and format of Switch- 
on sessions. Closer monitoring by the developers could help to ensure greater fidelity in future. 
It was also reported that some class teachers had limited awareness of Switch-on. Clarifying 
the role of Switch-on as part of a broader, teacher-led literacy strategy could improve 
implementation. 


Interpretation 


Primary analysis: the effect of Switch-on on reading outcomes 

Our effectiveness trial did not find evidence that a ten-week delivery of Switch-on improves reading 
outcomes of participating pupils more than schools’ usual practices. This finding runs counter to 
indications from the academic literature on high-intensity one-on-one interventions (D’Agostino and 
Murphy, 2004; Elbaum ef al., 2000; Jun et al., 2010; Slavin et a/., 2011) and is not in agreement with 
findings from a previous efficacy trial (Gorard et. al., 2014). 


Possible reasons for not finding an effect if Switch-on was effective 

Our evaluation gives no indication that Switch-on affects outcomes of eligible participating pupils. While 
we believe this result to be robust, our trial was limited to detecting effects of 0.25 standard deviations 
80% of the time. This limitation is a statistical property that applies, with different cut-offs, to all 
randomised controlled trials. If Switch-on had a true effect of less than 0.25 standard deviations, the 
likelinood of detection are even smaller than 80%. 


Furthermore, compensatory behaviour by control schools may have reduced our ability to detect an 
effect. Compensatory behaviour occurs if control schools, as a result of being assigned to the control 
condition, implement additional reading support. Our finding that 46% of treatment schools, but only 
25% of control schools, implemented no other literacy intervention (apart from Switch-on in treatment 
schools) indicates that some compensatory behaviour may have taken place. To the extent that these 
additional interventions were effective, the trial’s ability to detect the true effect of Switch-on would be 
diminished. 


Finally, the fact that post-intervention outcomes were measured approximately one month after the end 
of the intervention may not have provided sufficient time for improved reading skills to become apparent. 


43 


Switch-on Effectiveness Trial 


Possible reasons why Switch-on was less effective in this effectiveness trial 

It is also possible that Switch-on was effective at smaller scale (as measured in the efficacy trial) but is 
not effective at the size of implementation in this trial. Three findings indicate that this may be a 
contributing factor. 


First, the process evaluation indicates that, in some schools, class teacher involvement in Switch-on 
was limited. While part of the appeal of Switch-on as an intervention is that it is TA-led, evidence from 
the literature indicates that one-to-one catch up interventions should be ‘additional but explicitly linked 
to normal teaching and that teachers should monitor progress to ensure tutoring is beneficial’ (Higgins 
et al., 2014, p. 10). This suggests that even if TAs deliver the intervention, class teachers should be 
involved to ensure that the intervention is embedded in a wider literacy strategy for the child. The fact 
that class teachers were not systematically included in the planning and delivery of Switch-on in schools 
could therefore have undermined the achievement of outcomes. This is a challenge common to many 
one-to-one interventions that require children to leave the classroom. The previous efficacy trial 
highlighted the challenge of scheduling the intervention within the school timetable as a barrier to 
successful implementation. This remained a challenge in the current effectiveness trial and the 
inconsistent involvement of class teachers appears to have exacerbated timetabling difficulties. 


Second, the literature identifies having a strong champion at a senior level as a success factor (for 
example, Tanner et a/., 2015) for effectively implementing literacy interventions. Evidence from the 
process evaluation suggests that the engagement of senior level staff was inconsistent across 
participating schools. This meant resource and timetabling issues were not always resolved adequately, 
which affected schools’ ability to deliver the intervention effectively. 


Third, there is evidence from the process evaluation that schools undertook some level of adaptation 
and modification of the content and format of the intervention. This evidence reflects the findings of the 
previous efficacy trial, where some staff strayed from the Switch-on protocol and thus potentially 
reduced the intervention’s impact. While the efficacy trial nonetheless found improvements to reading 
outcomes, the ‘train the trainer’ model used in this trial may have led to greater changes to content and 
format. Examples of adaptations identified in this trial include changing the number of texts covered in 
a session, changing the duration of a session, or leaving out the writing component in Switch-on 
Reading and Writing. In addition, in response to lack of space, some schools had to deliver the 
intervention in busy, common areas in the school rather than a quiet space suitable for one-to-one 
delivery without interruptions. This is likely to have affected the quality of delivery. Previous evaluations 
of one-to-one catch-up literacy interventions, including the previous efficacy trial of Switch-on, have 
reported on school infrastructure having a bearing on successful implementation (Tanner et a/., 2011; 
Gorard et. al., 2014; NFER, 2014). 


In addition to potential challenges around implementation at a larger scale, the difference in estimated 
effect between efficacy and effectiveness trials could also be due to differences in the trial design and 
study population. For example, the efficacy trial focused on Year 7 pupils while our trial focused on Year 
3 pupils. It may be that Switch-on works better for older pupils. 


Also, in the pupil-level randomised efficacy trial, non-participating eligible pupils acted as controls with 
an assumption that non-participation has no effect. However, if non-participation had a negative effect 
on pupils with low reading skills in schools that delivered Switch-on (as we have found indicative 
evidence for), a pupil-level randomised trial would over-estimate the true effect of Switch-on. Instead of 
measuring the difference between the effect of Switch-on (say, A) and ‘business as usual’ (zero), it 
would measure the difference between the effect of Switch-on on eligible pupils (A) and the effect of 
Switch-on on non-eligible pupils (-B). The caution to this interpretation is that in our trial there are likely 
to be differences between participating and non-participating pupils as schools made decisions about 
participation. In the efficacy trial, participation was selected by lottery, so treatment and control pupils 
were similar. Furthermore, the intervention effect on non-participating eligible pupils was not a primary 
focus of the study and thus needs to be interpreted with caution. 
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Secondary analyses - the effect of Switch-on on pupils not receiving the intervention 

The unexpected findings from the secondary analysis raise the possibility of unintended consequences 
from implementing Switch-on. In particular, our trial found a moderate statistically significant negative 
effect of Switch on eligible pupils that did not receive the intervention. Findings from the process 
evaluation indicated that the majority of school co-ordinators perceived that, because of TA involvement 
in Switch-on, other pupils received less support than they would have otherwise. This suggests that the 
removal of TAs may negatively impact the results of pupils that were previously supported by those 
TAs, which indicates that careful planning of TA deployment is critical for a successful delivery of Switch- 
on. This finding regarding other under-performing pupils is not in line with our implicit theory of change, 
which hypothesised that a positive effect from Switch-on would have a positive spill-over effect on other 
pupils with lower reading standards. Furthermore, as indicated in earlier sections of our report, this 
finding would need to be considered as indicative due to the large proportion of missing data and the 
increased risk of finding spurious effects when carrying out multiple hypothesis testing. 


Another unexpected result from the secondary analyses indicated that delivering Switch-on has a small, 
positive effect on pupils whose reading ability does meet age-expected standards. This does not lend 
support to our original hypothesis—that positive effects from Switch-on would raise the standards of 
low-achieving pupils and have a lesser, but positive, spill-over effecton adequately achieving 
pupils. We can speculate that the removal of under-performing pupils from the classroom had a positive 
effect on better performing pupils because of increased teacher support and availability. However, as 
with other results from secondary analysis described above, this finding would need to be interpreted 
with caution due to the issues around missing data and multiple hypothesis testing. 


Limitations 


First, the change from a one-sided to a two-sided statistical test, and the low correlation between 
baseline scores and outcomes, resulted in a reduction of statistical power compared to our original 
estimates. The intervention would have to have had an effect of 0.25 standard deviations for our trial to 
have a chance of at least 80% of detecting the effect. 


Second, the differential loss to follow-up in the three core trial arms reduces the confidence we have in 
our findings. The support from the multiple imputation analysis for our findings is conditional on the 
assumption, implicit in any multiple imputation and not verifiable, that the missingness is random, 
conditional on factors for which we have data. 


Thirdly, data was not collected systematically about which classes eligible participating pupils missed 
because of their involvement in the intervention. The process evaluation indicates varying practices in 
schools, with some delivering Switch-on during literacy lessons and others withdrawing pupils from 
other lessons. This could be an explanatory factor that means any positive impact of Switch-on is 
counter-balanced by reduced participation in normal literacy lessons, which the study has not been able 
to investigate in detail. 


Future research and publications 


Future research should examine further the likelihood of effectiveness of one-to-one interventions 
delivered by TAs, in different contexts, subjects, age ranges, that are delivered over a longer period of 
time and with a greater follow-up period. Given the cost of the intervention, it may be advisable to 
determine the minimum effectiveness that the intervention would have to show to be cost-effective. 


Our findings also highlight the risk of negative spill-over effects on non-participating low-performing 
pupils, and these should be examined in any educational intervention that reduces resources for such 


pupils. 
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Appendix A: EEF cost rating 


Cost ratings are based on the approximate cost per pupil per year of implementing the intervention over 
three years. More information about the EEF’s approach to cost evaluation can be found here. Cost 
ratings are awarded as follows: 


Cost rating | Description | 


FELEEL Very low: less than £80 per pupil per year. 
FELEL Low: up to about £200 per pupil per year. 


FELEL Moderate: up to about £700 per pupil per year. 
LLLEE High: up to £1,200 per pupil per year. 
£LELE | Very high: over £1,200 per pupil per year. 
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Appendix B: Security classification of trial findings 


Fai (lat-Wielme-LiLare| 


Fair and clear quasi- 
experimental design for 
comparison (e.g. RDD) with 
appropriate analysis, or 
experimental design with 
minor concerns about 
validity 


11-20% 


38 Well-matched comparison 
(using propensity score 
matching, or similar) or 
experimental design with 
moderate concerns about 
validity 
2a Weakly matched comparison 
or experimental design with | MDES< | 34. 4ooz 
major flaws 0.5 
14 : 
Comparison group with poor 
or no matching (E.g. MDES < 51-50% 
volunteer versus others) 0.6 
0a 
No comparator ar > <50% 


Adjustment 
for 
Balance 


[0] 


Adjustment 
for threats 
to internal 

validity 


[-1] 


Initial padlock score: lowest of the three ratings for design, power and attrition = 4 padlocks 
Reason for adjustment for balance (if made): None required as this is well- balanced 
Reason for adjustment for threats to validity (if made): The intervention was not 
implemented as planned as some schools charged for the intervention and some control 
schools had introduced a breakfast club. 
Final padlock score: initial score adjusted for balance and internal validity = 4 padlocks 
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Appendix C: Analysis plan 


Switch-on Analysis Plan 


Introduction 


This pre-analysis plan sets out the detail of the analysis planned for the cluster-randomised 
controlled effectiveness trial of Switch-on, a reading intervention. 


Switch-on is an intensive ten-week literacy intervention targeted at children not 
demonstrating age-expected levels at primary school. It aims to enable children to 
‘participate more fully in the classroom by becoming more confident, active and independent 
readers, who can use a range of effective reading strategies in order to achieve their full 
reading potential (Nottinghamshire County Council Switch-on training manual). Switch-on is 
delivered by specially trained teaching assistants (TAs) in daily 20-minute sessions. 


The trial runs two versions of the intervention, implemented through a scaled-up delivery 
model. Both versions aim to improve reading skills. However, Switch-on Reading includes 
only reading exercises, while Switch-on Reading and Writing combines the teaching of 
reading skills with a focus on writing. Otherwise the two interventions are the same: both 
have the same amount of TA-pupil contact, the same amount of TA training and deliver a 
similarly designed intervention. The delivery model minimises the involvement of the 
intervention developers by cascading the training of teaching assistants through trained 
trainers, so that this model more accurately reflects the conditions of scale up. 


Primary hypotheses 


The primary hypotheses (H1) is that pupils participating in Switch-on will have better reading 
outcomes than pupils not participating in the programme, but receiving business-as-usual 
support for their reading. The main hypothesis will compare the two interventions separately 
against the control. 


Secondary hypotheses 


Four secondary hypotheses will be examined, as set out below. These hypotheses will be 
considered preliminary and indicative. 


e H2: Pupils participating in Switch-on Reading will have different (better or worse) 
outcomes than pupils participating in Switch-on Reading and Writing. 


e H3: Non-participating eligible pupils in the same class as pupils who participate in 
Switch-on Reading or Switch-on Reading and Writing will have better reading outcomes 
than eligible pupils in classes without pupils who participate in either Switch-on 
programme. 


e H4: Non-participating ineligible pupils in the same class as pupils who participate in 
Switch-on Reading or Switch-on Reading and Writing will have better reading outcomes 


than ineligible pupils in classes without pupils who participate in either Switch-on 
programme. 
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e H5: Switch-on Reading and Switch-on Reading and Writing (separately considered) will 
have a different (higher or lower) impact on pupils participating in the intervention and 
eligible for free school meals. 


Sample 


The 184 schools participating in this trial were recruited by the 21 trainers whose role it is to 
train the teaching assistants. Each trainer aimed to recruit nine schools in their local area in 
England'’, although some trainers only successfully recruited eight schools. 


No selection or exclusion criteria were set on school type, size, denomination, etc., but 
schools with an existing whole-school implementation of Reading Recovery principles were 
ineligible. Trainers were also asked to prioritise recruiting schools with higher levels of 
deprivation. 


Participating pupils include all pupils in Year 3 of recruited schools (9332 pupils aged 7 and 
8 years, of which 202 opted out). Pupils fall into two groups: ‘core-trial pupils’ were 
designated, prior to randomisation, to receive the intervention if the school was allocated to a 
treatment condition. If the school was allocated to the control group, the allocated pupils 
formed part of the core trial control group. ‘Spill-over pupils’ were not pre-allocated to receive 
the intervention and were included in the trial to measure potential spill-over effects. 


To select core-trial pupils, each school designated, prior to randomisation, between two and 
four teaching assistants that would deliver Switch-on if the school was assigned to a 
treatment condition. Each school then matched each teaching assistant with between two 
and four eligible pupils to receive the intervention, so that in each school between four and 
16 pupils were designated to receive the intervention subject to the randomisation. 


The trial thus includes all pupils that schools have capacity to teach, as they would in a non- 
trial setting, and thus all eligible pupils when eligibility is conceptualised as including schools’ 
capacity constraints. 


Power calculations 


For the primary hypothesis test, Table 16 presents the MDES when comparing Switch-on 
Reading only with the control (the same MDES would apply for comparing Switch-on 
Reading and Writing with the control). All calculations assume 80% statistical power and 
0.05 statistical significance for a two-tailed test. 


The table presents a range of MDES for different assumptions about the intraclass 
correlation coefficient (ICC), i.e. the degree to which intervention effectiveness clusters 
within schools, and for the predictive power of covariates. Estimates from recent studies 
show that ICC values of 0.1 are frequently obtained, but 0.2 presents the conservative 
estimate. Adjusting analysis for baseline outcomes reduces heterogeneity in outcomes and 
thus increases power. Bloom (2007) estimates that these range from 0.12 to 0.31 for 
individual and from 0.18 to 0.73 for school level variance explained by individual level 
baseline measures two years prior to the intervention. For simplicity, the same value was 
assumed for both individual and school level variances in the table below. 


The MDES thus ranges from 0.15 in the most optimistic scenario (ICC of 0.1 and 60% of 
variance explained by covariates) to 0.26 (ICC of 0.2 and no variance explained by 


'3 The trial took place in seventeen local authority areas: Birmingham, Bristol, Buckinghamshire, Cambridgeshire, 
Cheshire, Coventry, Derbyshire, Northamptonshire, Kent and Medway, Lincolnshire, Stoke on Trent, Sandwell, 
Walsall, Warrington, Wiltshire and Yorkshire 
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covariates). For our study, we are assuming an ICC of 0.15 and an A’ of 0.4 as conservative 
but realistic scenario, with an MDES of 0.22 standard deviations. 


Table 16: Minimum detectable effect sizes: SO Reading only 


Intraclass correlation 
0.10 0.15 0.20 

- + 0.0* 0.27 0.29 0.30 
o a0 
5 8 xe} 2 * 
oe ® © 0.2 0.24 0.26 0.27 
8 = fs S 0.4* 0.21 0.24 
ae: | : 7 

0.6* 0.17 0.18 0.19 


Randomisation 


The randomisation was carried out independently by NatCen Social Research. Schools were 
randomly assigned to Switch-on Reading, Switch-on Reading and Writing or the control 
condition using a stratified randomisation. The single stratification factor was ‘trainer’, 
indicating the trainer employed by the programme that trains teaching assistants in a 
particular geographical region. This was done in order to ensure a set number of treatment 
schools per trainer. Where the number of schools linked to a trainer was not divisible by 
three, the remainder was allocated by simple random allocation. 


Data 


Outcome 


The outcome measure for the trial is raw reading score, as measured by the Hodder Group 
Reading Test (HGRT). 


Data sources 


The analysis will use data from four sources, as set out in Table 17: 


Table 17: Data sources 


Source Data 
School baseline survey 


e Participating core-trial and spill- 
over pupils, such as their name, 
Unique Pupil Number (UPN) and 
date of birth 


e Information on why pupils were 
selected for the Switch-on 
intervention 
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Data on disability and SEN status 
in an anonymised, school- 
aggregate format 


Registers Information on take-up, drop-out 
and dosage of each individual 
core-trial pupil 

Pupil test Reading outcome — Hodder Group 


Reading Test 


National Pupil Database (NPD) 


pupil Key Stage 1 reading scores, 
as baseline outcome 


Pupil level information: 


Establishment number of the 
school attended as assigned by the 
DfE. 


Age of pupil at start of the 
academic year (in full years). 


Month part of age of pupil at start 
of the academic year. 


Deprivation indicator - IDACI score 


Is pupil known to be eligible for 
FSM? 


Flag to indicate if pupil has been 
recorded as eligible for free school 
meals in any termly School 
Census, AP Census or PRU 
Census in the last 3 years 


Flag to indicate if pupil has ever 
been recorded in any Spring 
Census as being eligible for free 
school meals 


Flag to indicate if pupil has been 
recorded as eligible for free school 
meals in any termly School 
Census, AP Census or PRU 
Census in the last 3 years 


Flag to indicate if pupil has ever 
been recorded in any Spring 
Census as being eligible for free 
school meals 


Gender. 


Institution type generated by AAT. 
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Reading attainment point score. 
Writing attainment point score. 


Average Reading and Writing 
attainment point score 


Number of sessions possible for 
the academic year. Based on 
termly sessions for all schools 
except special schools for which 
annual sessions data will be used. 


Number of sessions missed due to 
authorised absence during the 
academic year. Based on termly 
sessions for all schools except 
special schools for which annual 
sessions data will be used. 


Number of sessions missed due to 
unauthorised absence during the 
academic year. Based on termly 
sessions for all schools except 
special schools for which annual 
sessions data will be used. 


School level information: 


Establishment number as assigned 
by the DfE. 


Number of Pupils 
Number of school places collection 


Indicator showing whether a school 
is in an urban or rural area 


Percentage of pupils known to be 
eligible for FSM 


Percentage of pupils with special 
needs with statements 


Percentage of pupils with special 
needs without statements 


IDACI Band 


Percentage of pupils classified as 
white British ethnic origin 


Percentage of pupils whose first 
language is known or believed to 
be other than English 
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e Pupil teacher ratio 
e Full time Pupils 


e Total full time equivalent of 
teaching assistants 


e Total Pupils at level 2 or above in 
reading 


e Total Pupils in reading 


e Denominator for APS - all pupils 
reading 


Analysis 


Baseline characteristics 


Baseline characteristics will be summarised for core-trial pupils by intervention group across 
schools and pupils. Variables available at pupil level will be presented at that level, otherwise 
at school level. 


Continuous variables will be summarised with descriptive statistics (n, mean, standard 
deviation, range and median). 


At school level, the comparison will cover: 

e School type 

e Urban/rural status of schools 

e English as additional language (EAL) status 
e Special Educational Needs (SEN) status 


At the pupil level, the following baseline comparisons will be presented: 
e FSM eligibility 

e Gender 

e Birth term 

e KS1 reading outcome 


A summary of baseline characteristics will be presented in the additional analysis section of 
the report, covering the two spill-over pupil groups. 


Trial completion 

CONSORT diagram will be used to present summary of the flow of core-trial pupils and their 
schools from recruitment through baseline assessment, randomisation, post intervention 
assessment and analysis. The number of pupils and schools included or excluded at each 
stage will be clearly stated and the reasons for exclusion will also be stated. 


In addition, the flow of spill-over analysis pupils will be presented separately. 
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Main hypothesis 


The main and preliminary analyses will be conducted on an intention-to-treat basis and will 
include all core-trial pupils pre-designated to receive treatment subject to randomisation 
status for which outcome data has successfully been collected. 


The preliminary analysis will compare the effects sizes between the two treatment arms. If 
there is no statistically significant difference, the main analysis will merge the two 
intervention groups and consider them jointly against the control. If a statistically significant 
difference is found, the two groups will be compared separately against the control. 


The model analysed will be a random effects model which allows for school effects on 
reading attainment: 


Reading;; = Bo + 6, baseline;; + up; + uy ;baseline;; + e;; 


where / presents the school and u; is the group effect. In line with the EEF Analysis 
guidance, other covariates will not be taken into account at this stage. 


The calculation of effect sizes will follow the methodology for Hedges’ g, using the 
differences in means between treatment and control groups as numerator and the pooled 
standard deviation based on the total variance from a multilevel model as denominator. The 
effect size calculation will be based on Hedges’ (2007) approach assuming unequal sample 
size per school. 


The analysis will run in R version 3.2.1, and at its core use EEF’s analytical package for 
education trials, eefAnalytics, in the latest available version. 


No adjustment will be made for multiple comparisons, as the outcomes of the two 
interventions are considered to be highly correlated and findings in one intervention is likely 
to support the finding in the other (Schulz and Grimes, 2005). 


Sensitivity analyses 


A range of sensitivity analyses will be carried out to explore the robustness of the main 
finding. The following analyses will be carried out: 


e Including a wider range of prognostic covariates to increase power: Including school- 
level proportions of pupils with SEN and disability, school-level urban/rural indicator and 
individual-level indicators of English as Additional Language, free school meal eligibility, 
gender and birth term, as additional co-variates; 


e CACE analysis: We will label any (core-) trial participants that has not participated in at 
least 50% of sessions as non-compliant and estimate a complier average treatment 
effect (CACE); 


e If baseline imbalance is found, an analysis adjusting for baseline imbalance will be 
carried out. 


Secondary analyses 


Secondary analyses will explore hypotheses H3 — H5 set out above. All analyses will include 
the full set of predictive covariates set out in the sensitivity analysis. 


Hypothesis H3 is a spill-over analysis that explores whether pupils who were eligible but not 
selected for Switch-on were impacted by the participation of their peers. This may occur, for 
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example, because these pupils may now receive less support from their teaching 
assistant(s). The analysis will merge Switch-on Reading and Switch-on Reading and Writing 
on the assumption that any spill-over effects arise through the use of teaching assistants in 
an intensive intervention rather than due to the intervention content. 


Hypothesis H4 is an extension of the spill-over analysis H3, and covers pupils not eligible to 
receive Switch-on support. It will be carried out in the same way as H3. 


Hypothesis H5 is a requirement for analysis in all EEF trials and explores the impact of the 
intervention on pupils eligible for free school meals. The analysis will be comparable to the 
main analysis, but include all predictive covariates (in order to increase power) and merge 
Switch-on Reading and Switch-on Reading and Writing (also for power reasons). 


Attrition 


We expect an overall low rate of attrition. At the individual level, we expect to lose around 
10% of pupils due to moves and other external factors influencing participation in the final 
outcome testing. Given that we include only a database-driven baseline measure and no 

other co-variates in the main analysis, we do not expect further losses due to lack of data. 


If loss-to-follow-up can be predicted using existing covariates, multiple imputation will be 


used to infer the likely results of those lost to follow-up, and these imputed data will be 
included in the analysis. 
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Appendix D: Randomisation syntax 


name: <unnamed> 
log: I:\Workdocs\Tenders\1Tenders with an outcome\Won\EEF\I11798 Switch On 
Evaluation\Secure 
> \Data management \randomisation-log.txt 
log type: text 
opened on: 21 Oct 2015, 16:20:19 


set more off 


import excel using "School IDs.xlsx", sheet("School IDs") firstrow clear 
case(lower) 


. // set up 
- g sch = respondent_id 


. egen tra = group (trainer), label missing 
- g _assignment = @ 
. label define _alb @"No assignment (error)" 1 "SO R" 2"SO RW" 3"Control" 


. label values _a _alb 


. //checks 
. * Duplicate schools 
. duplicates 1 sch 


Duplicates in terms of sch 


(®@ observations are duplicates) 


* Number of schools per trainer 
* There must be between 8 and 10 schools per trainer 
. ta tra,m 


group(trainer) | Freq. Percent Cum. 


Trainer @1 | 
Trainer 92 | 
Trainer @3 | 
Trainer 04 | 
Trainer @5 | 
Trainer 06 | 
Trainer Q7 | 
Trainer 8 | 
Trainer 99 | 
Trainer 10 | 
Trainer 11 | 
Trainer 12 | 
Trainer 13 | 


OOUWMWDAAUAWMHW AW WO 
BHRRRRHRRRHRRHRHHA 
oO 
io 
Ww 
N 
fon) 
py 
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Trainer 14 | 9 4.89 65.22 
Trainer 15 | 18 5.43 70.65 
Trainer 16 | 9 4.89 75.54 
Trainer 17 | 9 4.89 80.43 
Trainer 18 | 9 4.89 85.33 
Trainer 19 | 9 4.89 90.22 
Trainer 20 | 9 4.89 95.11 
Trainer 21 | 9 4.89 100.00 
eseeewe ee eee ew ew ee ee ee SS eee eet tie ee ee eet eell ell lela ll ia 
Total | 184 100.00 


. // randomisation 
. * seed and random number assignment 
set seed 1206364607 //from random.org 


. di "Seed: " %10.0f c(seed) 
Seed: X7eeb8b23c43F462544a474abacbdd93d00042570 


. g double r = @ 


replace r = runiform() 
(184 real changes made) 


. * Assign three schools to Group 1, three schools to Group 2, and the rest to Group 
3. 
sort tra r 


. by tra: replace _a = trunc((_n+2)/3) 
(184 real changes made) 


replace _a=3 if _a>3 
(1 real change made) 


. // save output 
sort tra _a sch 


. drop sch trar 


. export excel using "Schools-assignment.xlsx", firstrow(var) replace 
file Schools-assignment.xlsx saved 


. log close _all 
name: <unnamed> 
log: I:\Workdocs\Tenders\1Tenders with an outcome\Won\EEF\I11798 Switch On 
Evaluation\Secure 
> \Data management \randomisation-log.txt 
log type: text 
closed on: 21 Oct 2015, 16:20:19 
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Appendix E: Memorandum of Understanding for school 


Vhe Education 
-@-  NatCen eee 


Social Research that works for society Foundation 


Switch-on Effectiveness Research Project 


We are very pleased and grateful that your school wishes to take part in our study to find out if Switch- 
on can be an effective national intervention. 


Two versions of the intervention are being evaluated: 


e Group 1: Switch-on — reading and writing 
e Group 2: Switch-on — reading only. 


Your school will be allocated to one of these groups or a control group (Group 3). See Section 4 below 
for more information. 


This document will help to clarify what is involved and act as a memorandum of understanding between 
your school and Switch-on. We will be working with the research team from NatCen Social Research 
(NatCen) who will conduct an independent evaluation of the intervention. 


1. Aims of the project 


The aim of the project is to test the effectiveness of the Switch-on Reading and Writing intervention in 
accelerating the literacy learning of pupils. The target group is underperforming Year 3 children working 
below age related outcomes at the end of Key Stage 1. 


2. About the teams 


The Education Endowment Team 

The Education Endowment Foundation (EEF) is an independent grant-making charity dedicated to 
breaking the link between family income and educational achievement, ensuring that children from all 
backgrounds can fulfil their potential and make the most of their talents. The EEF have provided the 
funding for this project, following a successful smaller trial of the Switch-on approach. 


The Switch-on team 

A Switch-on trainer will train and support each school in Group 1 or Group 2. This trainer will be the first 
point of contact for all enquiries or concerns relating to the programme. 

The Switch-on developers will train the trainers and provide resources for the schools in Group 1 and 
Group 2. They will coordinate the project and be available for trainer and evaluator enquiries. 


The NatCen Team 


NatCen is an independent social research organisation that carries out research on a wide range of 
social issues. The NatCen evaluation team are independent of the Switch-on team and have been 
funded by the Education Endowment Foundation (EEF). Our evaluation will aim to gain a better 
understanding of how the programme is working in practice, gather the views of trainers and school 
staff and assess the impact on pupil attainment in literacy. The evaluation will report on the feasibility 
of rolling the programme out to more schools. 
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NatCen will be doing all of the data collection and analysis. They will also liaise directly with your school 
in order to 


¢ Provide you with the parent information letter to be sent out. The purpose of this letter will be 
to: 
o seek parental consent for their child to be involved in the study 
o take the tests required for the evaluation 
o allow NatCen to link their child’s information with test results and the National Pupil 
Database. 
« Provide you with links to the 2 online surveys. 
¢ Liaise with your school about the testing and support arrangements for invigilators to be 
present at the testing sessions. 
¢« To seek access to the school and relevant staff, to take part in interviews and case studies 
(more information can be found in the process evaluation section of this document) 
« Respond to any queries you may have about the research tasks or the research itself. 


3. Rationale 


Previous research for Switch-on Reading, 
www.educationendowmentfoundation.org.uk/projects/switch-on-reading/, has shown 
that the intervention has a positive impact on literacy learning among Year 7 pupils, particularly pupils 
with SEN, those in receipt of pupil premium and lower attainers. The 2015-2016 effectiveness study will 
investigate: 


1. The impact of the Switch-on (reading and writing) intervention on the literacy learning of 
underperforming children working below age related outcomes in Year 3 

2. The impact of the Switch-on Reading intervention on the reading abilities of underperforming 
children working below age related outcomes in Year 3 

3. If the Switch-on (reading and writing) intervention has a greater impact on children’s reading 
ability than Switch-on Reading alone 

4. If these interventions have an impact on secondary outcomes, such as, self-esteem, attitude to 
and engagement in learning 

5. If Switch-on training and support develop the knowledge, skills and understanding related to 
literacy learning of Teaching Assistants (TAs) in primary schools 


The research findings will provide valuable information about the elements of the intervention that have 
a positive effect on pupils and practitioners. Based on the evaluation findings, adjustments can be made 
to the training of trainers and practitioners to enable Switch-on to be rolled out on a wider scale. 


4. Evaluation design 


Participating schools will be randomly allocated to either: 
1. GROUP 1 - to deliver Switch-on (Reading and Writing) with 4-16 targeted pupils 
2. GROUP 2 - to deliver Switch-on Reading with 4-16 targeted pupils 
3. GROUP 3 - to continue business as usual with targeted pupils and receive a financial incentive 
that can be used at a later date to purchase Switch-on training. These schools will form the 
‘control group’ for the evaluation. 


Random allocation of schools is the best way to study the impact Switch-on has on children’s literacy 


attainment. It is important that schools understand that even if they are allocated to GROUP 3 (control 
group) they will still need to undertake all evaluation activities they are asked to do. 
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5. Data sharing 


All the information that we gather about individual pupils, teachers and schools will be kept completely 
confidential in accordance with the Data Protection Act. No information about individual children will be 
made available to anyone outside of the research teams at Switch-on, NatCen and the Education 
Endowment Foundation (who fund the work). 


Data Archiving — Fischer Family Trust will collate and anonymise the data for upload to the UK Data 
Archive. The archived data will be available in an anonymised form with restricted access for research 
purposes only 


Evaluation report - NatCen will not use pupils’ or teachers’ names or the names of any of the schools 
involved in the project in the final report. The accounts of the effectiveness of Switch-on will be 
presented in the form of aggregated or averaged data. 


6. Requirements for NatCen’s Evaluation 
i. Background survey: 


At the beginning of the project, schools will be asked to provide school and pupil level information via 
an online ‘background’ questionnaire. This will include: 


e school’s Unique Reference Number (UCN) and LAESTAB code 

e names of all pupils in Year 3 with a flag for the children who receive Switch-on support 

e Unique Pupil Numbers (UPNs) for all Year 3 pupils 

e indicator if the pupil’s parent has opted out of the evaluation 

e information about TAs involved in the programme including their contact details and 
qualifications. 


The NatCen team will use this information and access the National Pupil Database (NPD) to collect 
data on relevant pupils in order to assess any impact of the project on attainment. 


ii. Process evaluation 


As part of the evaluation, the NatCen team will visit 6 of the schools involved to understand how Switch- 
on is delivered. Participating in these visits is voluntary. A NatCen researcher will arrange visits at a 
time convenient to the school. These visits will include an interview with the Switch-on co-ordinator, 
teaching assistants and with a classroom teacher. We will also view the delivery of one Switch-on 
session. This would take place in the spring term 2016. 


In addition, NatCen will also be inviting TAs to take part in telephone interviews (in those schools not 
involved as a case study). These interviews will also be voluntary and will be arranged at a time 
convenient to the TA. 

iii. Testing 
All schools (in groups 1, 2, and 3) would need to complete a test assessing the reading of every child 


in Year 3. The test would take place at the start of the summer term 2016 after the trial and would take 
approximately one hour to complete. It is recommended that this takes the place of a literacy lesson. 
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Schools will need to agree to testing dates and times in advance and allow for an invigilator to attend 
each session to ensure that the tests are completed consistently across all schools. 


iv. Implementation survey 


Following the trial, all schools (in groups 1, 2 and 3) will need to complete a second online survey. This 
would include details about the literacy interventions going on in your school during this period, and 
costs incurred. 


7. Responsibilities of all parties 
The Switch-on team will: 


e provide the evaluators with a list of participating schools 
e be the first point of contact for any queries about the programme. 


If the evaluator (NatCen) randomly assigns your school to Group 1, Switch-on will: 


* — supply school with all resources including 2 Switch-on book packs (£1,800) and sets of teaching 
resources (£200). 

* provide 1 ¥ days initial training in November 2015 

* provide a school visit for support in early February 2016 

* provide a follow-up half day training in late February 2016. 

The training will be for 2-4 teaching assistants and a teacher from the school management team who 

will be the project coordinator in your school. 


If the evaluator (NatCen) randomly assigns your school to Group 2, Switch-on will: 


* supply school with all resources including 2 Switch-on book packs (£1,800) and sets of teaching 
resources (£100). 

* provide 1 day initial training in November 2015 

* provide a school visit for support in early February 2016 

* provide a follow-up half day training in late February 2016. 

The training will be for 2-4 teaching assistants and a teacher from the school management team who 

will be the project coordinator in your school. 


If the evaluator (NatCen) randomly assigns your school to Group 3, Switch-on will: 
* promptly pay the financial compensation for taking part at the end of the spring term 2016 when 
all Year 3 pupils have been tested. 


Schools 


GROUP 1 responsibilities: 

¢ send out parent opt-out letters to all Year 3 pupils asking for consent to participate in all aspects 
of the evaluation and collect any opt-out slips. The letter would be provided for you, you would 
just need to post this out to parents 

* complete a ‘background’ survey which would provide the research team with details about the 
pupils in the year, the pupils chosen to be part of the trial, pupils’ UPNs and background 
information about the school and the TAs involved 
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release one member of the senior management team (coordinator) and 2-4 teaching assistants 
for 1% days training in the autumn term 2015 (1 day for Switch-on Reading training 2 day for 
Switch-on Writing training) 

timetable the 2-4 teaching assistants to work for 20 minutes every day for 10 weeks with each 
target child (2-4 children for each teaching assistant) 

your school must start the intervention the week commencing the 4" January 2016 

fill in registers for each session 

release the coordinator for the ’2 day support visit from the Switch-on trainer in spring term 
release the teaching assistants for the 12 day support visit from the Switch-on trainer in spring 
term 

release the coordinator and 2-4 teaching assistants for 2 day training in the spring term 2016 
complete an ‘implementation’ survey after the trial has finished. This would include details about 
the other literacy interventions going on in your school during this period and information about 
how you have delivered Switch-on and the costs incurred 

the whole of Year 3 would need to complete either an online test to asses reading or a paper- 
based test assessing their reading and writing at the end of the trial. The test would take place at 
the start of the Summer Term 2016 and would take around an hour to complete. It will be 
recommended that this takes the place of a literacy lesson. 

the coordinator will support pupil testing by providing NatCen interviewers with access to schools, 
planning testing sessions and ensuring that all pupils are tested within the designated testing 
timeframe 

ensure the shared understanding and support of all school staff to the project and personnel 
involved 

be a point of contact for parents/carers seeking more information on the project. 


GROUP 2 responsibilities: 


send out parent opt-out letters to all Year 3 pupils asking for consent to participate in all aspects 
of the evaluation and collect any opt-out slips. The letter would be provided for you, you would 
just need to post this out to parents 

complete a ‘background’ survey which would provide the research team with details about the 
pupils in the year, the pupils chosen to be part of the trial, pupils’ UPNs and background 
information about the school and the TAs involved 

release one member of the senior management team (coordinator) and 2-4 teaching assistants 
for 1 day training in the autumn term 2015 (1 day for Switch-on Reading training) 

timetable the 2-4 teaching assistants to work for 20 minutes every day for 10 weeks with each 
target child (2-4 children for each teaching assistant) 

your school must start the intervention the week commencing the 4' January 2016 

fill in registers for each session 

release the coordinator for the ’2 day support visit from the Switch-on trainer in spring term 
release the teaching assistants for the /2 day support visit from the Switch-on trainer in spring 
term 

release the coordinator and 2-4 teaching assistants for 2 day training in the spring term 2016 
complete an ‘implementation’ survey after the trial has finished. This would include details about 
the other literacy interventions going on in your school during this period and information about 
how you have delivered Switch-on and the costs incurred 

the whole of Year 3 would need to complete either an online test to asses reading or a paper- 
based test assessing their reading and writing at the end of the trial. The test would take place at 
the start of the Summer Term 2016 and would take around an hour to complete. It will be 
recommended that this takes the place of a literacy lesson. 

the coordinator will support pupil testing by providing NatCen interviewers with access to schools, 
planning testing sessions and ensuring that all pupils are tested within the designated testing 
timeframe 


64 


Switch-on Effectiveness Trial 


* ensure the shared understanding and support of all school staff to the project and personnel 
involved 
* bea point of contact for parents/carers seeking more information on the project. 


Responsibilities if in the intervention Group 3: 

* send out parent opt-out letters to all year 3 pupils asking for consent to participate in all aspects of 
the evaluation and collect any opt-out slips. The letter would be provided for you, you would just 
need to post this out to parents. 

* carry out a ‘background’ survey which would provide the research team with details about the 
pupils in the year, the pupils chosen to be part of the trial, pupils’ UPNs and background 
information about the school and the TAs involved. 

* carry out an ‘implementation’ survey after the trial period has finished. This would include details 
about the English interventions going on in your school during this period. 

« the whole of Year 3 would need to complete either an online test to asses reading or a paper- 
based test assessing their reading and writing at the end of the trial. The test would take place at 
the start of the Summer Term 2016 and would take around an hour to complete. It will be 
recommended that this takes the place of a literacy lesson. 

« — the coordinator will support pupil testing by providing NatCen interviewers with access to schools, 
planning testing sessions and ensuring that all pupils are tested within the designated testing 
timeframe 

* ensure the shared understanding and support of all school staff to the project and personnel 
involved 

* bea point of contact for parents/carers seeking more information on the project. 


Schools musi not, under any circumstances, change the pupils allocated to the intervention 
once they have been entered into the background survey. 


NatCen 


NatCen will: 
e conduct the random allocation and inform schools 
e collate School- and Pupil- Level data provided by schools 
e obtain National Pupil Database data for participants from the DfE 
e work closely with the school coordinator to plan and invigilate testing 
e store all data safely and securely 
e analyse data from the project in order to produce impact estimates 
e conduct the process evaluation, including analysis and reporting from this 
e produce end of project evaluation report 
e provide head teachers with all attainment data after the tests have been completed 
e share the final report with the school 


If you have any questions about the research or concerns about your role and responsibilities, please 
contact us. You can telephone or email Paula Burrell paula.burrell@nottscc.gov.uk 
07771345930 or Jose Coles jose.coles@nottscc.gov.uk 07979744941. 


Once again, we would like to express our gratitude to you for joining us in this research. We believe 
that it will be a thoroughly worthwhile project and that it will produce some valuable results 
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We commit to the Evaluation of Switch-on as detailed above. As a school, | commit to remaining 
a part of the Switch-on Effectiveness research project as detailed above for the period of 
September 2015-April 2016 


Head teacher name: 


School name: 


School address: Postcode: 


School Tel no: 


Head teacher Signature: Date: 


Head teacher Email address: 


Switch-on co-ordinator name: contact 
number: 


Switch-on co-ordinator email address: 


Switch-on co-ordinator position at school: 


Number of Teachers (including SLT) at school as at [Insert date] 


Number of pupils at school as at [Insert date] 


Jose Coles and Paula Burrell — Switch-on developers 
Rakhee Patel and Sarah Haywood — NatCen Social Research 


Please complete and sign copies, retaining one and returning the second copy to your Switch-on 
trainer by September 18th 2015. 
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Appendix F: Post-intervention survey 


P11798 Follow-up Survey 
SPEC 


Questions are documented as follows: 
{Question routing- who is asked the question} 
Question Name 

Question text 


: Question response options 


If the respondent presses “next” without selecting an answer then please have “Don’t 
know” appear as an answer code option. 


{ASK ALL} 
Serial (VARLAB: Serial) 
Welcome to the final survey of the Switch-on Evaluation. 


Please enter the access code given to you by NatCen (this can be found in the email which 
contained the link to this information form). 


: Numeric (6 characters) 


Introduction 

{ASK IF=INTERVENTION SAMPLE} 

Dear {textfill coordinator name}, 

This follow-up information form is the last step of the Switch-on evaluation. 


It will help us to evaluate the impact of this intervention on the reading level of your pupils and 
to gather information on any cost incurred to you. 


Please fill this survey before May 20" 2016. 


As usual, if you have any questions whilst completing this form please contact Michael directly 
on 0808 281 0308 or email: switchon@natcen.ac.uk 
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{ASK IF=COMPARISON SAMPLE} 


Dear {textfill coordinator name}, 


Thank you again for taking part in the Switch-on trial. 
This follow-up information form is the last step of the Switch-on evaluation. 


All the information given to us will be kept securely and used for research purposes only. 


Please fill this survey before Wednesday 8th" June 2016. 

If you have any questions whilst completing this form please contact Michael Lumpkin directly 
on 0808 281 0308 or email: switchon@natcen.ac.uk 

{ASK ALL} 

Laestab (VARLAB: LA Number ) 

What is your school’s LA/Estab number? 


Your LA/Estab number is composed of 3 numbers followed by a forward slash and then 4 
numbers. This is an example: 961/4009 


If you are not sure of what yours’ is, you can find’ it here: 
http://www.education.gov.uk/edubase/home.xhtml 


: Numeric 


Hard check: [If incorrect / too few / too many characters] MGURDAESTapmbenislcomposea 
of 3 characters followed by forward slash followed by 4 characters. 


A - LITERACY INTERVENTIONS IN THE SCHOOL 


{ASK ALL} 
OtherInt (VARLAB: Whether other literacy interventions took place) 


From this list, please select all those interventions that took place in your school only during 
the Spring Term of 2016 (January-March) with your Year 3 pupils. 


{DISPLAY IF = INTERVENTION SAMPLE} 


If any other literacy interventions took place during that time with Year 3 pupils that are not 
listed here, please select “other”. If you had no other literacy intervention than Switch-on taking 
place in Year 3 please select “none”. 


{DISPLAY IF = COMPARISON SAMPLE} 
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If any other literacy interventions took place and are not listed here, please select “other”. If 
you had no literacy intervention taking place in Year 3 please select “none”. 


MULTICODE 


None 

Acceleread 

Better Reading Partnerships 
Boosting Reading 

Catch-up Literacy 

Direct Phonics 

Inference Training 

Nessy 

. Reading Recovery 

10. Toe by Toe 

11. Switch-on {ONLY SEE IF = COMPARISON} 
12. Other intervention(s) 


OHONAARWNS 


{ASK IF OTHERINT = 12} 
OtherNum (VARLAB: Number of other interventions) 


How many other literacy interventions did you have during the Spring Term of 2016 
(January-March) for your Year 3 pupils? 


‘Numeric {soft check between 1 and 5} 


{ASK IF OTHERINT = 12} 
Othercha (VARLAB: Characteristics of the other interventions) 


We would like to know more about the other literacy intervention(s) that took place in your 
school during the Spring Term of 2016 for Year 3 pupils. 


Please tick all the relevant options which best describe the intervention(s) that you delivered. 
:GRID with 

Rows = {feed forward from OtherNum response} 

Columns = 


1. Daily intervention 

2. Weekly intervention 

3. One-to-one reading support 

4. Small group reading and writing support 


MULTICODE so that 


- 1 and2 cannot be both selected for the same intervention 
- 3and4 cannot be both selected for the same intervention 
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B - FIDELITY TO PROGRAMME 


{ASK IF=INTERVENTION SAMPLE} 
TAdeliver (VARLAB: Whether the TA was replaced) 


Before Switch-on started, you told us that the following Teaching Assistants (TAs) would be 
delivering Switch-on. For each TA, please tell us if they were able to deliver the intervention 
until the end of the trial? 


‘Feedforward TAfirst and TAsurname from baseline survey and by each name please display 
as a drop-down menu the following options: 


1. Yes, this TA delivered the whole Switch-on intervention for ten weeks 
2. No, this TA withdrew from the intervention before the start of the trial 
3. No, this TA withdrew from the intervention during the trial 


{ASK IF=INTERVENTION SAMPLE AND TAdeliver = 2 or 3} 
TAwhy (VARLAB: Why TA did not deliver whole intervention) 


These are the TAs who did not deliver Switch-on until the end of the trial. For each TA, please 
tell us why they withdrew by choosing one of the options 


‘Feedforward TAfirst and TAsurname where TAdeliver = 2 or 3, and by each name please 
display as a drop-down menu the following options: 


Left the school 

Was re-allocated to another intervention 
Was re-allocated to other duties 
Long-term sick 

Other 


OV Go Noe 


{ASK IF=INTERVENTION SAMPLE AND TAdeliver = 2 or 3} 
TAreplace (VARLAB: Who replaced the TA) 
When a TA had to withdraw from the trial, what did you do? 


‘Feedforward TAfirst and TAsurname where TAdeliver = 2 or 3, and by each name please 
display as a drop-down menu the following options: 


Replaced the TA with another TA trained directly by a Switch-on trainer 

Replaced the TA with another TA not trained directly by a Switch-on trainer 

Replaced the TA with another member of staff 

Did not replace the TA and stopped the intervention for the pupils allocated to this TA 
Other 


oP ONS 
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{ASK IF=INTERVENTION SAMPLE} 
Fidelity (VARLAB: keys aspects of fidelity) 


We would like to know more about how the intervention was delivered. For each statement 
below, please tell us if this occurred always, most of the time, some of the time or never: 


:GRID 


1. The sessions were in a one-to-one setting 
2. The sessions lasted 20 minutes 


: Present the sentences in a grid and for each sentence, please display the following options 


Always 

Most of the time 
Some of the time 
Never 

Don’t know 


OPO NS 


{ASK IF=INTERVENTION SAMPLE} 
Daily {VARLAB: Did the sessions occur daily} 
Did the sessions occur daily? 


1. Yes 
2. No 
3. Don’t know 


{ASK IF=INTERVENTION SAMPLE} 
Replic (VARLAB: whether coordinator thinks training was followed) 


How closely do you think the guidance on how to deliver Switch-on was followed by TAs? 
Please think about the guidance at the face-to-face training days and in the Switch-on manual. 


Very closely 
Closely 

Not too closely 
Not closely at all 


PON > 


{ASK IF=INTERVENTION SAMPLE} 


Challg (VARLAB: what aspects of the project were challenges) 
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Below are a number of statements about potential challenges of implementing Switch-on. For 
each one, please indicate how much you agree or disagree. 


:GRID 


oP 
reas 


COON @ oP 6 


The pressures on our TAs’ time was a challenge 

Because of the TA’s involvement in Switch-on, other pupils received less support 
than they would have otherwise 

The training events and materials were not clear enough 

TAs didn’t feel confident about implementing the programme 

Doing all the paperwork related to the sessions was a challenge 

Finding a room for the sessions was a challenge 

The costs were a challenge for our school 

The content of the programme was not always suitable for the selected pupils 


: Please present the sentences in a grid, and for each sentence please display the following 


options 


OU 


Strongly agree 
Agree 

Disagree 
Strongly disagree 
Don’t know 


C - MEASURABLE COSTS 


{ASK IF=INTERVENTION SAMPLE} 


Matcosts (VARLAB: Material costs) 


For each of the items below, please tell us how much you, as a school, spent in total on each 
item throughout the trial. Please enter '0' if you have not spent anything on the following items. 


:GRID 


1. 


Additional books £ 


{Soft check: should not exceed 1000} 


2. Printing/photocopying = 


{Soft check: should not exceed 100} 


3. Travel (e.g. for TAs to get to training) e 


{Soft check: should not exceed 800} 


4. Other (please specify) = 
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Error message: “Are you sure? That sounds like rather a lot. Please check and amend if 
necessary." 


{ASK IF=INTERVENTION SAMPLE} 
TAcover (VARLAB: whether had to cover for TA time) 
Did you have to organise cover for your TA(s) during the Switch-on training day(s)? 


1. No cover was required 
2. Yes 


{ASK IF=INTERVENTION SAMPLE and TAcover= 1} 
TAcovhrs (VARLAB: whether had to cover for TA time) 
How many hours or days did you organise cover for? 


: Hours {Soft check if over 300 hours} 
: Days {Soft check if over 10 days} 


{ASK IF=INTERVENTION SAMPLE} 
Extrares (VARLAB: Whether had to purchase staff) 


Did you hire any new staff for the delivery of this project or extend the contracted hours of any 
existing staff? 


Note: We are not asking about employed staff members working unpaid overtime 


1. Yes - increased the contracted hours of existing staff 
2. Yes — employed somebody new 
3. No 


{ASK IF=INTERVENTION SAMPLE AND Extrares = 1} 
Extrahrs (VARLAB: Extra hours purchased) 


How much did you spend per month on additional hours for existing staff for this intervention 
from September 2015 to March 2016? 


: Numeric 


Soft check: [needs to be between 0 and 20,000 ] 
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{ASK IF=INTERVENTION SAMPLE AND Extrares = 2} 
Salcosts (VARLAB: Salary costs) 


How much did you spend per month on salary costs for new staff for this intervention from 
September 2015 to March 2016? 


: £Numeric 


Hard check: [needs to be between 0 and 4,000 | 


D. COSTS IN TERMS OF TIME 


{ASK IF=INTERVENTION SAMPLE} 
TAprep (VARLAB: total TA time preparing for Switch-on) 


Excluding the time spent attending the training and taking part in elements of the 
evaluation (such as doing an interview), please tell us how much time TAs spent preparing 
themselves and their materials before the intervention started in January (for example 
counting the 100 words in the books, photocopying their forms, practising Running Records, 
etc...) 


You can tell us in either days or hours. 
: Numeric (days) {hard check if over 20 days} 


: Numeric (hours) {hard check if over 140 hours} 


{ASK IF=INTERVENTION SAMPLE} 

TAtasks (VARLAB: daily TA time on tasks) 
Please tell us how many hours TAs spent daily: 
GRID 


e Preparing for each session 
e Doing tasks after the sessions 


1. : Less than 1 hour 
2. :Numeric {Hard check if over 5 hours} 


{ASK IF=INTERVENTION SAMPLE} 


COtime (VARLAB: Coordinator time spent on Switch-on tasks) 
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Thinking of your tasks and role as a Coordinator, how much time did you spend working on 
Switch-on in total from September to March? (not counting the tasks for the evaluation, i.e. 
the online information forms or anything related to the testing) 


You can tell us in either days or hours 
: Numeric (days) {hard check if over 140 days} 


: Numeric (hours) {hard check if over 980 hours} 


{ASK IF=INTERVENTION SAMPLE} 
Recomm (VARLAB: Would you recommend this intervention) 
Would you recommend the Switch-on intervention to another school? 


1. Yes 
2. No 
3. Don’t know 


{ASK ALL} 
ImprEv (VARLAB: How could we improve the evaluation process) 


Now we would like to ask you about your experience of the evaluation process. Is there 
anything that we could have done to make this process easier for you? 


‘text {maximum 4000 characters} 


{ASK ALL} 
ImprTest (VARLAB: How could we improve the testing process) 


Now we would like to ask you about your experience of the testing process: is there anything 
that we could have done to make this process easier for you? 


‘text {maximum 4000 characters} 


Thank you 
{ASK ALL} 
Thank you for your help. 


We have now collected all the information we need from your school for the evaluation of 
Switch-on. 
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If you are interested in the results of our study, our report will be publicly available on this 
website http://www.natcen.ac.uk/switch-on 


If you have any questions, please contact Michael directly on 0808 281 0308 or email: 
switchon@natcen.ac.uk 


Kind regards, 


The Switch-on Research team 
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Appendix G: Topic guides 


1. Coordinator topic guide 


Interviews with Switch-on coordinators 


Aim of the phone interview: 


The aim of the interview with school coordinators is to find out about their experience of 
the programme, the workload it created for them and how they liaised with other school 
staff and the Switch-on trainers for the delivery. Finally, it will explore their perceptions of 
the impacts of the programme. 


The topic guide: 


This guide sets out a number of topics and questions that will be covered during interviews. 
The guide does not contain follow-up probes and questions like ‘why’, ‘when’, and ‘how’, 
etc. as participants’ contributions will be explored in this way, as far as is feasible, during 
the 45 minute interview. Researchers will use prompts and probes in order to understand 
how and why views, behaviours and experiences have arisen. 


The interview will last for approximately 45 minutes. 


Interviewer: specify that the interview is about the intervention, not about the 
evaluation. 


1. Introductions 

e Introduce yourself and NatCen Social Research 

e Introduce the study: 
o Evaluation of Switch-on Programme 
o Commissioned by the Education Endowment Foundation 

e Overall project aims: 
o Tounderstand experiences of delivering Switch-on and in particular to 

understand barriers to delivery and what enables delivery 

o Tounderstand the benefits of the programme for pupils. 


e Interview is one of 6 interviews with Switch-on school coordinators. 
e The interview is not about the evaluation aspect (baseline survey and co) but 
about the delivery. 


Digital recording — check OK, and reassure re: confidentiality 
Data kept securely in accordance with Data Protection Act 
How we'll report findings — anonymity of all participants 

Any questions/concerns? 
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2. Background 


Aim: To gather background information on the Switch-on Coordinator and the school. 


Current occupation — role in the school, ... 
Professional background 
Experience of delivering/ overseeing literacy interventions and role 
School context 
o Profile of pupils (FSM, EAL,SEN) 
How many TAs in the school 
Processes for inducting and training TAs; how they are deployed 
How much of a priority is improving reading in the school? 
Other reading programmes’ interventions currently or previously used 
in school 
e Involvement with Switch-on: 
o How first heard about the programme? 
o How was the decision to join the programme made? 
o How was the coordinator role assigned? 
o What did they know about it? 
What expectations did they have about the programme and their role 


3. Role in the Switch-on intervention and workload 


Aim: To provide understanding of what role the coordinators had in the Switch-on 
programme. 


What role in the programme? 
o Had a say in the selection of the pupils and of the Teaching Assistants 
o Had a Say in organisation and content of sessions 
o Administrative tasks 
o Other 
What type of contact with Switch-on trainer? 
o Frequency of contact 
o Format of contact 
o Content of communication 
o Other 
How they worked with TAs 
o Format of contact 
o Content of communication 
o Other 
How did Switch-on affect their workloads? 
o Any additional work required (beyond what they would normally do) 
o Was the workload manageable related to the tasks that the intervention 
implied: selecting the pupils and the TAs, contacting the trainer, attending 
the training...) 


4. Views on facilitators and barriers to implementation 


Aim: To provide an understanding of experiences of delivering Switch-on and lessons learnt 
from these experiences. 


What were the main challenges to implementation 
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o For staff 
o For the school 
o For pupils 
e What enabled implementation 
o For staff 
o For the school 
o For pupils 


e Crucial factors for successful implementation? 
e Views on whether it is replicable - why/why not? 
e Any schools in which it wouldn't work? 


5. Impacts and perception of benefits 
Aim: To establish the perceived impact on all pupils and staff. 


e Impact of the intervention on the Switch-on pupils 
o ability to read 
ability to write 
pupils’ wellbeing 
confidence in reading 
interest in reading 
o any other impact 
e Impact of the intervention on non-recipient pupils 
o Impact on the class as a whole? 
o ability to read 
o confidence in reading 
o interest in reading 
o any other impact 
e Main benefits of the intervention 
o For recipient pupils 
o For non-recipient pupils 
e Negative impacts of the intervention? 
e Unexpected consequences 
o Hidden costs 


ooO°0 0 
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2. Teaching assistants topic guide 


Interviews with Switch-on teaching assistants 


Aim of the phone interview: 


The aim of the interview with TAs is to explore how they were trained to deliver Switch-on 
and their experience of delivering the ten week programme. The interview will ask TAs 
about their views on what enables or acts as a barrier to successful implementation and 
perceived impacts on pupils. 


The topic guide: 


This guide sets out a number of topics and questions that will be covered during interviews. 


The guide does not contain follow-up probes and questions like ‘why’, ‘when’, and ‘how’, 
etc. as participants’ contributions will be explored in this way using prompts and probes in 
order to understand how and why views, behaviours and experiences have arisen. 


The interview will last for approximately 45 minutes. 


Interviewer note: emphasize that the interview and observation are not about 
checking how well they are delivering Switch-on but rather to understand what they 
think about it and how they are implementing it. 


2. Introductions 

e Introduce yourself and NatCen Social Research 

e Introduce the study: 
o Evaluation of Switch-on Programme 
o Commissioned by the Education Endowment Foundation 

e Overall project aims: 
o Tounderstand experiences of delivering Switch-on and in particular to 

understand barriers to delivery and what enables delivery 

o To understand the benefits of the programme for pupils. 


e Interview is one of 20 interviews with Switch-on Teaching Assistants. 
e Repeat that the aim of the interview is not to evaluate the TAs’ performance, 
but to collect their thoughts on Switch-on. 


Digital recording — check OK, and reassure re: confidentiality 
Data kept securely in accordance with Data Protection Act 
How we'll report findings — anonymity of all participants 

Any questions/concerns? 


2. Background 


Aim: To gather background information including detail of the context in which the school 
operates and the profile of their pupils. 
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e Professional and educational background (e.g no. years as a TA, training, 
qualifications, experience of other literacy interventions) 
e Duties and role as a TA within school 
e Profile of Year 3 pupils: 
o Special Education Needs (SEN) 
o English as an additional language (EAL) 
o Disadvantaged children (based on FSM take-up) 
o Turnover of pupils at the school 
e Does the school have educational priorities? How much of a priority is 
improving reading in the school? 
o Other reading programmes/ interventions currently or previously used in 
school 
e Involvement with Switch-on: 
o Howfirst heard about the programme 
o How was the decision to join the programme made 
o What did they know about it 
o What expectations did they have about the programme and their role 


3. Training received from the Switch-on Trainers 
Aim: To provide understanding of how TAs were trained to deliver the intervention 


e Overview of the training 
o Nature and format + hours 
o Received any material for the intervention? 
o Content — probe: was comprehension a focus? 

e Any other preparation before intervention, such as practising using the forms, 

readings, meetings, scheduling, ... 

o Nature of this 
o Support received from the school coordinator and Year 3 staff 
o Support received from trainer 
o Support from others 

e Thoughts and expectations about the intervention 

e Effectiveness of training/other preparation 
o Which aspects of the training/preparation helped most? 
o Which helped least? 
o How prepared they felt to deliver Switch-on 
ie) 

4. Programme-delivery 


Aim: To provide understanding of how the intervention was delivered in practice, if this 
differed from the guidance provided and reasons for this, and what impact being part of 
Switch-on had on their workload 


e Selecting pupils 
o How were pupils selected to receive the intervention? 
o Who had a say, who made the final decision? 
o Onhindsight are they satisfied with this process and outcome of the 
selection? 
e Format of the sessions 
o How was organised: time set aside in timetable? Ad hoc? 
o Regularity 
o Format 
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o Location 
o Attendance and pupil engagement 
e Have they done the Switch-on sessions the way they had planned? 


o Format 
o Using the documents and materials provided 
e Materials 


o What materials did they use? 
o Whether materials targeted at the right age and abilities 
o Suggested improvements to materials and forms 
e Monitoring progress 
o How was pupil progress monitored 
e Communication with the Class Teacher 
o In regards to content of the intervention 
co Inregards to monitoring and sustaining progress 
e Differences of delivery to different subgroups (E.g. high ability, low ability, middle 
ability, FSM, ESL) 
o How this was managed within the intervention 
o Was the programme adapted for particular groups 
o Impact on the planning and delivery of the session 
e TA workload: session replacing other duties or on top? 
o Time spent planning — including any joint planning with teachers 
o Time spent delivering 
o Unexpected tasks 
o Could they cope with that workload all the time? 
e Did delivery of Switch-on go as planned? Any changes related to: 
o pupils 
TAs 
mode/format of delivery 
dosage 
content 
o anything else 
e What aspects of delivery worked well 
e What aspects of delivery worked less well 


oOo°0 0 


5. Support received and contact with the coordinator and trainer 
Aim: To establish what support TAs received from Switch-on trainers and coordinators 


e Contact with Switch-on trainer 

o Who initiated it and why — regular check-in or when there were specific 
issues? 
How often 
Format 
Content of contact: help, clarification, support? 
Any additional support would have liked 
Anything they would have liked to change 
e Support from Switch-on Coordinator? 

o What kind of support and for what 
How often 
Format 
Content of contact: help, clarification, support 
Any additional support would have liked 
Anything they would like to have changed 


(e) 


00000 
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Mid-way visit 
o What happened 
Format 
Views on content 
Views on helpfulness 
Any improvements or other visits they would have liked 


oOO0°0 0 


6. Impacts 


Aim: To provide an understanding of outcomes of the programme on different groups of 
pupils, to gather information about the positive and negative impacts. 


Were there benefits for intervention pupils, and if so why? 
Probe 
o ability to read 
reading comprehension 
ability to write 
confidence in reading 
interest in reading 
pupils’ perception of their own abilities 
pupils’ wellbeing 
o Any other impact? 
Were there benefits for the rest of the Year 3 pupils and why? 
Probe 
o Impact on the class as a whole, did it change the culture around reading? 
ability to read 
ability to write 
confidence in reading 
interest in reading 
pupils’ perception of their own abilities 
pupils’ wellbeing 
Any other impact? 
Negative impacts of the intervention? 
Unexpected consequences 
o Hidden costs or time needs 


C0000 0 


(e) 


7. Overall views and suggested improvements 


Aim: To gather views on the programme as a whole, whether they would recommend it and 
how it can be improved. 


If there were benefits, what helped to achieve these benefits for the pupils? 
o How/ Why? 
What prevented progress/benefits for pupils? 
o How/ Why? 
Any improvements 
o Materials (folders, forms, books) 
o Support structures 
o Programme itself 
Recommend use to other schools 
o Why/ why not? 
Views on whether Switch-on would appeal to other schools 
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6. Overall views and suggested improvements 


Aim: To gather views on the programme as a whole, whether they would recommend it and 
how it can be improved. 


e Does the programme meet its aims 
o How/ Why not? 
e Any additional support they would have liked 
o At what stage? 
o In what form? 
o Relationship with Switch-on 
o Within school (from Year 3 teacher, headteacher) 
e Any improvements 
o Materials 
o Support structures 
o Programme itself 
e Recommend use to other schools 
o Why/ why not? 
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3. Class teacher topic guide 


Interviews with class teachers on Switch-on 


Aim of the phone interview: 


The aim of the interview with classroom teachers is to explore their experience of the 
programme, the workload it created for them, how they liaised with other school staff for 
the delivery and their perception of the impact that the programme had. 


The topic guide: 


This guide sets out a number of topics and questions that will be covered during interviews. 
The guide does not contain follow-up probes and questions like ‘why’, ‘when’, and ‘how’, 
etc. as participants’ contributions will be explored in this way, as far as is feasible, during 
the 45 minute interview. Researchers will use prompts and probes in order to understand 
how and why views, behaviours and experiences have arisen. 


The interview will last for approximately 45 minutes. 


1. Introductions 
e Introduce yourself and NatCen Social Research 
e Introduce the study: 
o Evaluation of Switch-on Programme 
o Commissioned by the Education Endowment Foundation 
e Overall project aims: 
o Tounderstand experiences of delivering Switch-on and in particular to 
understand barriers to delivery and what enables delivery 
o Tounderstand the benefits of the programme for pupils. 
Interview is one of 6 interviews with classroom teachers. 
Digital recording — check OK, and reassure re: confidentiality 
Data kept securely in accordance with Data Protection Act 
How we'll report findings — anonymity of all participants 
Any questions/concerns? 


2. Background 


Aim: To gather background information including previous experiences of teaching and of 
working on literacy interventions. 


Current occupation — role in the school. 
Professional and educational background 
Experience with other literacy interventions 
Involvement with Switch-on: 
o Howfirst heard about the programme? 
How was the decision to join the programme made? 
What did they know about it? 
What expectations did they have about the programme and their role 


oOo Oo 
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3. Role in the Switch-on intervention and workload 


Aim: To provide an understanding of the role that class teachers had in the Switch-on 
intervention . 


e What role in the programme? 
o Selection of the pupils 
o Deciding on how Switch-on would be delivered in the school (timetabling, 
format of sessions, content) 
o Administrative tasks 
o Other 
e How did they work with TAs? 
o Joint planning 
o Did they communicate with TAs on the pupils’ experience of the 
intervention 
o Sharing information to monitor improve pupils’ progress 
o Supporting TAs during the ten-week delivery period 
e What did TAs find difficult or need support with? 
e How did Switch-on affect teachers’ workloads? 
o Any additional work required (beyond what they would normally do) 
o Was the workload manageable? 
e Views on the content of the programme 
o Suitability for different kinds of pupils (EAL, SEN, confident/shy, 
behavioural issues) 
o Aligned with needs 
o Aligned with curriculum 
e Views on the format of the programme 
o Suitability for different kinds of pupils 
o Aligned with needs 
o Aligned with curriculum 
e Did delivery of Switch-on go as planned? Any changes related to: 
o pupils 
TAs 
mode/format of delivery 
dosage 
content 
anything else 


O © Or OF © 


4. Impacts and perception of benefits 


Aim: To provide an understanding of outcomes of the programme on different groups of pupils, 
to gather information about the positive and negative impacts. 


e Were there benefits for intervention pupils, and if so why? 
e Probe 
o ability to read 
ability to write 
confidence in reading 
interest in reading 
pupils’ perception of their own abilities 
pupils’ wellbeing 


OO)" 0.0! 20. 
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o Any other impact? 
e Were there benefits for the rest of the Year 3 pupils and why? 
e Probe 
o Impact on the class as a whole, did it change the culture around reading? 
ability to read 
ability to write 
confidence in reading 
interest in reading 
pupils’ perception of their own abilities 
pupils’ wellbeing 
o Any other impact? 
e Main benefits of the intervention 
o For recipient pupils 
o For non-recipient pupils 
e Negative impacts of the intervention? 
e Unexpected consequences 
o Hidden costs 


000000 


5. Views on facilitators and barriers to implementation 


Aim: To provide an understanding the various experiences of implementation and lessons 
learnt from these experiences. 


e What were the main challenges to implementation 
o For staff 
o For the school 
o For pupils 
e What enabled implementation 
o For staff 
o For the school 
o For pupils 
e Crucial factors for successful implementation? 
e Views on whether it is replicable - why/why not? 
e Any schools in which it wouldn’t work? 


6. Overall views and suggested improvements 


Aim: To gather views on the programme as a whole, whether they would recommend it and 
how it can be improved. 


e Does the programme meet its aims 
o How/ Why not? 
e Any additional support they would have liked 
o At what stage? 
o In what form? 
o Relationship with Switch-on 
o Support from within school (headteacher, coordinator, SENCO) 
e Any improvements 
o Materials 
o Support structures 
o Programme itself 
e Recommend use to other schools ? Why/ why not? 
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4. Trainers topic guide 


Interviews with Switch-on trainers 


Aim of the phone interview: 


The aim of the interview with trainers is to explore how they were trained and how they 
themselves trained TAs to deliver the intervention. The interviews will explore how well the 
schools engaged in the programme and the workload that this implied for the trainers. 
Finally, the interview will ask about the facilitators and barriers to successful 
implementation and views on the impacts of the programme on pupils and schools. 


The topic guide: 


This guide sets out a number of topics and questions that will be covered during interviews. 
The guide does not contain follow-up probes and questions like ‘why’, ‘when’, and ‘how’, 
etc. as participants’ contributions will be explored in this way, as far as is feasible, during 
the 45 minute interview. Researchers will use prompts and probes in order to understand 
how and why views, behaviours and experiences have arisen. 


The interview will last for approximately 45 minutes. 


1. Introductions 
e Introduce yourself and NatCen Social Research 
e Introduce the study: 
o Evaluation of Switch-on Programme 
o Commissioned by the Education Endowment Foundation 
e Overall project aims: 
o Tounderstand experiences of delivering Switch-on and in particular to 
understand barriers to delivery and what enables delivery 
o Tounderstand the benefits of the programme for pupils. 
e Interview is one of 5 interviews with Switch-on trainers. 


Digital recording — check OK, and reassure re: confidentiality 
Data kept securely in accordance with Data Protection Act 
How we'll report findings — anonymity of all participants 

Any questions/concerns? 


2. Background 


Aim: To gather background information including previous experiences of delivering training, 
and of working on literacy interventions 


Current occupation — where based, job(s), 
Professional and educational background 
Experience in education, literacy interventions, and delivering training 
Experience working with schools and TAs 
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e How got involved with Switch-on: how heard about it, what did they know 
about it. 
e Understanding of the Switch-on intervention and of their role before training 


3. Training received from Switch-on 


Aim: To provide understanding of how the trainers were trained to deliver the Switch-on 
intervention and to train the TAs 


e Training delivered by Switch-on 
o Format and nature of the training, hours, location, materials received 
o How effective this was - what was most useful / what was least useful 
o How prepared they felt to implement the programme 
e Preparing to train the TAs 
o Process of being allocated schools and TAs 
o Communication with TAs and coordinators 
o Organising the training 
e Support from Switch-on developers 
o Nature of the support received 
o Frequency of contact 
o Mode of contact, ease of asking for support 
o Suggested improvements to the training received and communication with 
developers 


4. Training the TAs 


Aim: To provide understanding of how the TAs were trained, how consistent the training 
was and what worked best. 


e Preparation required 


e Delivery 
o Was guidance given by developers followed? Reasons for any 
changes. 
o Attendance and impact of absences 
e Materials 


o What materials they used 
o What materials they provided TAs 
o Suggested improvements to the materials used and provided 
e Engagement of the schools, TAs and coordinators 
o Different levels of initial engagement from schools and TAs 
o What did TAs find easy 
o What did TAs find hard 
o Which sort of TAs struggled 
e Overall experience of training TAs 
o Difficulties and barriers experienced 
o Most positive elements 
o Lessons learnt from the training experience 
e Support received from Switch-on in preparing and delivering the training 
o Nature and frequency of support 
o Adequate level of support? 
o Suggested improvements to the support 
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5. Programme-delivery (engagement with schools + workload) 


Aim: To establish how the programme was delivered, what were the different levels of 
need, engagement and communication with the participating schools? 


e Engagement with schools once the intervention began 
o General overview of engagement with schools 
o How easy/difficult trainers found this process 
o Any characteristics of easy vs difficult schools to engage with 
e Type of support provided 
o What type of support TAs asked for 
o What type of support coordinators asked for 
o Any general rule regarding who would ask for support 
o Any common difficulty that required support 
e Half intervention visit 
o What the purpose was 
o Did it meet the purpose — why/why not? 
e Form of communication with schools 
o How often would communicate: only when issue or check-ins 
o What format? Emails, phone calls, texts, visits 
e Workload 
o What was workload compared to expectations 
o How managed workload 


6. Facilitators and barriers to implementation 


Aim: To provide an understanding of experiences of delivering Switch-on and lessons 
learnt from these experiences. 


e Main challenges and barriers to implementation 
o Working with schools and staff 
o Working with Switch-on developers 
e What worked well 
o Working with schools and staff 
o Working with Switch-on developers 
e Examples of schools that engaged well and delivered Switch-on effectively 
o Characteristics of the school and staff involved 
o What made a difference 
o Is it replicable 


7. Overall views and suggested improvements 


Aim: To gather views on the programme as a whole, whether they would recommend it 
and how it can be improved. 


e Does the programme meet its aims 
o How/ Why not? 
o Views on benefits for pupils 
o Views on benefits for TAs 
o Views on benefits for schools 
e Any additional support they would have liked 
o At what stage? 
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o In what form? 
o Relationship with Switch-on 
o Relationship with schools 
e Any improvements 
o Materials 
o Support structures 
o Programme itself 
e Recommend roll-out to other schools 
o Why/ why not? 


91 


You may re-use this document/publication (not including logos) free of charge in any format or medium, 
under the terms of the Open Government Licence v2.0. 


To view this licence, visit www.nationalarchives.gov.uk/doc/open-government-licence/version/2 
or email: psi@nationalarchives.gsi.gov.uk 


Where we have identified any third party copyright information you will need to obtain permission from 
the copyright holders concerned. The views expressed in this report are the authors’ and do not 
necessarily reflect those of the Department for Education. 


This document is available for download at www.educationendowmentfoundation.org.uk 


N Education 


Endowment 
Foundation 


The Education Endowment Foundation 

9th Floor, Millbank Tower 

21-24 Millbank 

London 

SW1P 4QP 
www.educationendowmentfoundation.org.uk 


